examples/Playing Atari with Deep Reinforcement Learning (OpenAI Gym).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agentnet intro\n",
    "\n",
    "In this tutorial we'll\n",
    "* еrain on Atari MsPacman game with deterministic framerate using OpenAI gym\n",
    "* train a simple lasagne neural network for n-step q-learning\n",
    "* no experience replay, 10 synchronous agents, soft target networks\n",
    "\n",
    "_Note that this agent is not state of the art on this particular game_, it's just simple and it works.\n",
    "\n",
    "\n",
    "### About OpenAI Gym\n",
    "\n",
    "* Its a recently published platform that basicly allows you to train agents in a wide variety of environments with near-identical interface.\n",
    "* This is twice as awesome since now we don't need to write a new wrapper for every game\n",
    "* Go check it out!\n",
    "  * Blog post - https://openai.com/blog/openai-gym-beta/\n",
    "  * Github - https://github.com/openai/gym\n",
    "\n",
    "\n",
    "### New to Lasagne and AgentNet?\n",
    "* We only require surface level knowledge of theano and lasagne, so you can just learn them as you go.\n",
    "* Alternatively, you can find Lasagne tutorials here:\n",
    " * Official mnist example: http://lasagne.readthedocs.io/en/latest/user/tutorial.html\n",
    " * From scratch: https://github.com/ddtm/dl-course/tree/master/Seminar4\n",
    " * From theano: https://github.com/craffel/Lasagne-tutorial/blob/master/examples/tutorial.ipynb\n",
    "* This is pretty much the basic tutorial for AgentNet, so it's okay not to know it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=device=gpu0,floatX=float32\n"
     ]
    }
   ],
   "source": [
    "#setup theano/lasagne. Prefer GPU\n",
    "%env THEANO_FLAGS=device=gpu0,floatX=float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#If you are running on a server, launch xvfb to record game videos\n",
    "#Please make sure you have xvfb installed\n",
    "import os\n",
    "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 10\n",
    "SEQ_LENGTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-04-03 01:35:45,336] Making new env: MsPacmanDeterministic-v0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f0f5552fad0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGvBJREFUeJztnXusVdWdxz8/eSPyurR4QSqkYUrRRKBU2/FRW7TDYFNt\nqkaNlYyMpA041umk1aFNbVIakmma2kokUB1RrNRaM9KGtLbMWGOtjpdqFQULU+XlBa6C4OOKUH7z\nxzmLe86559yz9157n7325fdJbu7da/3WWt+99l37t9djry2qimEYyTkpbwGGUXSsERmGJ9aIDMMT\na0SG4Yk1IsPwxBqRYXhijcgwPMmsEYnIXBF5WUS2icgtWZVjGHkjWUy2isgA4C/AxcAu4BngalV9\nKfXCDCNnBmaU79nANlX9K4CIrAUuBeo2ouHDh+vo0aNxDbr2d56ISNXvSkLSGSKN6q6yvkKou1qd\n7ndnZ+frqvqBZumzakQTgZ0Vx7uAcyoNRGQhsBBg5MiRLFiwgL/97W8AHDlyBOD4cZ4MGDAAgEGD\nBvWKC0lniDSqO1dvEEbd1ep0x9/97ne3R0mf1ePc5cBcVf3n8vGXgHNUdXE9+9mzZ2tHRwcHDx4E\n4K677gLgzTffTF1bXEaPHg3AggULesWFpDNEGtWdqzcIo+5qdY4aNQoAEdmoqrObpc9qYGE3MKni\n+LRymGH0O7J6nHsGmCoiUyg1nquAa5olOnr0KAC7du0C4PXXX89IXnTGjRsH9GirJCSdIdKo7ly9\nQRh119c1jkImjUhVj4rIYuA3wADgblV9MYuyDCNvsvJEqOp6YL1PHq6DN2bMmKrwAwcO9LJtZuMb\nn4bOuPFx8qjtoCfVlIbuEOouyf9IUmzFgmF4kpknSoORI0cCcP3111eFr1q1qpdtMxvf+DR0xo2P\nk0ftXTWppjR0h1B3Sf5HkmKeyDA8yWSeKC5unuiNN94AYOnSpUDP3XX8+PFV9nv37u2VRzObpPHu\nOXrJkiW9yoyrM258nDwa9YmyLLNZfKO6c/UG2dVdnP+RWp1tbW1A/vNEhnHCEHSfyN1dX3vttaa2\nzWx84/siqs40NETVmZamtPJoRIh1FxfzRIbhiTUiw/Ak6Me5KAwcWDqFsWPHAj0d6ka4x4f9+/cD\nyZd6xCGuxlApQt3VaqzE5eGoZ5ME80SG4UnhPZGbrFu0aBHQ+25Ti7v73H777VXHWRJXY6gUoe5q\nNVbi8nDUs0mCeSLD8KTwnuikk0r3AXfHci9UNcI9x7t0rSCuxlApQt31pdHlUZu3L+aJDMOTwnsi\nX9zoz6RJk6rCd+7cWc88szyKiNVdCfNEhuHJCe+JTjnlFACuuOKKqvA4y+TTyKOIFKHuKhfmdnZ2\n1o3znbczT2QYnpzwnsht07V8+fKq8O7ubiDanE4aeRSRItSdyz/LMswTGYYnJ7wnOnbsGADvvPNO\nrnkUkSLUncs/yzLMExmGJ4X3RG6Getu2bQB0dXX1ae+ekaOuQK7Nv15c2hpDJa26i7P6O+vrmwbm\niQzDk6A3Kglhi1mj/+O2EbaNSgwjJ6wRGYYn1ogMw5OgR+eGDh0KwIwZM6rCn3vuuV62zWzSjg+t\njPfee68qPGrd5XFeeVy/qDZJME9kGJ4E7YmGDBkC9L6DbN68uZdtM5u040Mro9YTRa27PM4rj+sX\n1SYJ5okMw5PE80QiMgm4FxgPKLBSVW8XkbHAz4DJwKvAlara59eUbJ7IyJM854mOAl9T1enAJ4BF\nIjIduAXYoKpTgQ3lY8PotyRuRKraqap/Kv/9FrAZmAhcCqwum60GLvMVaRghk0qfSEQmAzOBp4Hx\nqurew91D6XHPMPot3o1IREYAvwC+qqqHKuO01OGq2+kSkYUi0iEiHUVd1WwY4DnELSKDKDWg+1X1\n4XLwXhFpV9VOEWkH9tVLq6orgZVQGliAnmHZ888/H4C33nqrKo0bxu3o6OgV1gg36Th7dnX/0OWR\nJL0La0StzrhlNMs/K7LWnef160un2yzF/f/FJbEnEhEB7gI2q+oPKqLWAfPLf88HHklahmEUAR9P\ndC7wJeAFEXHrJ/4dWAY8KCILgO3AlVEzHDFiBABf+MIX6sa7IfAtW7YcD2t2J3J5fvGLX6wKd3kk\nSe+GQBtRqzNuGc3yz4qsded5/eLojEviRqSqTwDSIHpO0nwNo2jYigXD8MQakWF4EvQC1Ci4Z955\n8+ZVha9fv74l6UMpIwuKUDch1K15IsPwpPCeyC2grR2libqw1jd9KGVkQRHqJoS6NU9kGJ4U3hO5\nrWHXrVtXN37YsGGZpo9CK8rIgiLUTQh1a57IMDyxRmQYnlgjMgxPgugTdXV1sWLFiqZ2hw8fBnqv\n7s4SV9aaNWuOhzVb7RtXZ20ZcVYTu08lnnzyyVXhrq9Q+bnFZmStO8/rB8nqNwrmiQzDkyA80dtv\nv82TTz6Zt4y6uLvnxo0bgyxj8uTJAMycOROAk04q3Rc3bdoEwK5du1JQWJ9W1I0vTiNkp9M8kWF4\nEoQnMuLj+kDTpk0DYMeOHUDP8357ezsABw6Udis70T6F2UrMExmGJ0F7otGjRwNw3XXXVYXfe++9\nvWyj2PiUURsfxcY3vi8bt3p59+7dQM9Gl65P9MEPfhCAiy++GCj1O33LzCo+qk090vgfefPNNyOV\n1QjzRIbhSdCeqLu7G4Df//73dcMraWTjdnLxLaM2PoqNb3xfNu68pk6dCsCsWbOq7I4cOQLA448/\nDvTMl4RwXnlcv6g2STBPZBieWCMyDE+CfpxzE2V//vOfm9o2smn2OBC1DB8NacVX2riBhQkTJlQd\nO9xAgvsanDv2KTOr+L5s0rp+UW2SYJ7IMDwJwhOJyPGFlH3hXvk9duyYd5muvCjlxiWpTjc8Xdpc\nNloZL7zwQlVah3td2v3Osn7j6I5LmtcnK53miQzDkyA8UXt7O9/85jeb2rlJsdWrV/cKi4qbnFu8\neDEAR48ejZU+CrU6m2l0mubPn1913GpC1D1w4MCqvJNeb4iv85577olkZ57IMDwJwhMNGTKED3/4\nw03t3Ibo7u6UBPeMffrppyfOoxlxdTo791pD3hvaF013X1SeS1Y6zRMZhidBeCIf3IjLyJEjq8IP\nHTpUzzwX4mqsHB1zy3XSGJGsp6mveZj+WLeZaGhZSYbRTym8Jxo1ahQAN9xwQ1X4qlWr8pBTl7ga\nDx48ePzvO++8E+h5uS4txowZA8BXvvKVhjb9sW6zwDyRYXji7YlEZADQAexW1c+JyFjgZ8Bk4FXg\nSlVN9zZagesz/PSnP60bntecSyVxNVb2f9zLdu53WrittPrqa/XHus2CNDzRTcDmiuNbgA2qOhXY\nUD42jH6LlycSkdOAS4ClwL+Wgy8FLiz/vRp4DPiGTzl94VYcZLk1lC9F0FiPIugOQaOvJ/oh8HWg\n8plgvKp2lv/eA4yvl1BEFopIh4h0dHV1ecowjPxI7IlE5HPAPlXdKCIX1rNRVRWRul9bUtWVwEqA\nGTNmqJstr8TNAbgRmCS4Z//9+/cnziMurqw4W/gWkaLWresHulFQ3zk4n8e5c4HPi8g8YCgwUkTW\nAHtFpF1VO0WkHdjnpdAwAidxI1LVW4FbAcqe6N9U9VoR+Q9gPrCs/PuRZnl1dnaydOnSXuFjx44F\nYNGiRUllHp+5vuOOO4Bs3h+qxd0lQ5rZz4Ki1q3zQMuXLwf8PWkW80TLgItFZCtwUfnYMPotqaxY\nUNXHKI3CoapvAHPipD969Gif8yA+z6zuzpX2jL9R3Lp1/0/OA/nOwdmKBcPwpPBr59z7IqeeempV\n+J49e6qO047PoozQaFXdtqqMrDBPZBieFN4Tufdhrrnmmqrw2lW8acdnUUYS3Fza8OHDq8Lfffdd\nwK8/2aq6bVUZWWGeyDA8sUZkGJ4U/nHOTZytXLmyKrx2Mi7t+DTLcJPKSfjQhz4EwMc+9jGg5/Hu\nj3/8I+C3MLNVdZtlGT51GxXzRIbhSeE9kes4N9vUL+v4tPKIivtm6xlnnAHAzp07ARg8eDDQs9G9\nzzdbW1W3rSojK8wTGYYnQXiiRps3us003KRfki1/XdqJEydWHWeJ0+m+p5rFVsVuU3b3tfBG4Vls\nMu8oat3Wbjrp86oNmCcyDG+C8EQTJkzgW9/6VsN4N+JU78W9ZriNKm688UYAxo0bl0BhPNyCxmXL\nllUdZ4HrG9V+s9V91OvZZ5/NrOyi1q3zPO6jBo247bbbIuVnnsgwPAnCE0Hvj1SljXthLOtyKstq\nBe51hNrXpbPohzWiqHWbll7zRIbhSTCeKCluFGratGlV4Vu2bMlDTl2y0Og8j/uwce0duru7G+hZ\niJqEE7Vu42KeyDA8KbwnGjZsGACf+tSnqsK3b9+eh5y6xNVYOd8yZcoUoGfOrJYjR45U/Xa4+aFG\nH09zI2uVZdX2o/pj3WaBeSLD8KTwnsitmfrRj35UN74VcxfNiKuxcgbdzcG0gtp5uP5Yt1lgnsgw\nPLFGZBieWCMyDE+C6BO9/fbbPPnkk03t3Ieb3nvvvcRlHT58GIBNmzZVHadJXJ3OrqOjA+j7Y8SN\nqJ19V9Wq31HIQ3cz3DzQmWeemSh95blkpdM8kWF4EoQn6urqYsWKFS0py61uvv/++4FsV1hHpVZT\nHNwq7ssvvxyA999/H4CtW7cCPSsaslhL56M7Km50bcmSJYnSO42QnU7zRIbhSRCeyEiO2zth27Zt\nAFxwwQVAT1/g+eefz0fYCYR5IsPwJGhP5J7358yp/lLLhg0betlGsfEpozY+io1vfJw8XN/nrLPO\nAnq2FZ47d25mZaYVH9WmHmn8jyTZCakS80SG4YmXJxKR0cBPgDMBBa4HXgZ+BkwGXgWuVNVEX4Fy\nK5GHDh1aN7ySKDY+ZdTGR7HxjY+Th3u/yM0LuXmjEM8rj+vnU0YzfD3R7cCvVXUacBawGbgF2KCq\nU4EN5WPD6LckbkQiMgq4ALgLQFXfV9U3gUuB1WWz1cBlviINI2QkzrKQqoQiM4CVwEuUvNBG4CZg\nt6qOLtsIcMAdN6KtrU0vueSSRDqa0Wiyzn2tPITJ1jRwneYrrrgCgLVr1wLw2GOP5SUpFfK8fvfd\nd99GVZ3dzM7ncW4gMAu4U1VnAu9Q8+impRZat5WKyEIR6RCRjizWrxlGq/AZWNgF7FLVp8vHD1Fq\nRHtFpF1VO0WkHdhXL7GqrqTkyWhra0vmDo3jQ9mTJk0C4NFHHwXgiSeeyE3TiUZiT6Sqe4CdIvKR\nctAcSo9264D55bD5wCNeCg0jcHwnW28E7heRwcBfgX+i1DAfFJEFwHbgymaZtLW1cd111zUtzC0m\ndHfbyrCsGDFiBACf/exne8U5Hc00NMojavp6ebhjN0zrNhxxG5DUfsO0tu6SlBlXd7P0UXX44DRU\n6qgM64v77rsvkp1XI1LV54B6Ha/e0+CG0U8JYtnPyJEjueiii5rauY00Hn/88eNhWd/J3ATd+eef\n3yvO6WimoVEeUdPXy6Otra1pmkpq6y5JmY60zjuqDh8qJ1iT1l0zbNmPYXhijcgwPLFGZBieBNEn\n8sFtdHj11VdXhT/wwAMtSZ+lhrg64pBGmSHXXVb1Vg/zRIbhSeE9kVsy5F5Kqw0fNGhQpumz1BBX\nRxzSKDPkusuq3uphnsgwPCm8J3Ibcjz11FN145vNTvumb4WGqDrikEaZRai7tOutHuaJDMMTa0SG\n4Yk1IsPwJKg+0aFDhwBYs2YNAAcPHqyKd59UdB92ioKzXb58ed3wolB7HnFHnZLUXQj4Xr9Ku0Z1\n5+aarr32WqC0ljMO5okMw5OgPJG7W7otcdN4f95t5O7yLCr95Tzi4nvelRv5N8rD7eNQ+/HoqJgn\nMgxPgvJEtbgNCN1n1h3d3d29bJvZ+ManpTNO+jg6jx07lkhTlDIb4XvePjqTXD8fnX1hnsgwPAna\nE7lRkxtuuKEqfNWqVb1sm9n4xqelM076ODoPHKjeqTmqpihlNsL3vH10Jrl+Pjr7wjyRYXhijcgw\nPAn6cc59zfrnP/953fBKmtkkjXdbUPnobJZHo/RxdMbV1Ci/SpukutOsu0Y6k9SLj86+ME9kGJ4E\n7YncRNkrr7zS1LaZjW98X8TR6Zs+ahlR88zzvOPkkcZ5+OjsC/NEhuFJ0J4oCm6yzm3sPmDAgD7t\n3Rfl3n33XaD3JGUWuDLcAtva8CjEPc9akpx3Grp9KcL1NU9kGJ4U3hO5Zetf/vKXARgzZkyf9m5S\ncsWKFUBrXg1wd/La5fy1d/i+iHuetSQ57zR0+1KE62ueyDA8Kbwncp8UGT9+PNB8s/LBgwdXpWsF\nbgRq37663zuLRNzzrCXJeaeh25ciXF/zRIbhSdCeyI3EuLuQY+/evUGVETWPuPFxdcShkeYouuLG\nJ9GR1nlXjuZlVYaXJxKRm0XkRRHZJCIPiMhQERkrIr8Vka3l3/F6wIZRMBJ7IhGZCPwLMF1Vu0Xk\nQeAqYDqwQVWXicgtlD6G/I0kZbiRGbeBhCOtJexplRE1j7jxcXXEoZHmKLrixifRkdZ5V246klUZ\nvn2igcAwERkIDAdeAy4FVpfjVwOXeZZhGEGT2BOp6m4R+T6wA+gGHlXVR0VkvKp2ls32AL0fuiPi\n5iPuvvvuuuEQf74kbhlR8o+iM0l8XB1xaKQ5iq6o8WnUXVrXN8syEnuicl/nUmAKMAE4WUSq/KWq\nKqAN0i8UkQ4R6ejq6koqwzByx2d07iLgFVXtAhCRh4G/B/aKSLuqdopIO1B3kkFVVwIrAWbPnl23\nobl1UGlsndWINMqImodvfJrEOe8sdWd9jV3+WZbh0yfaAXxCRIaLiABzgM3AOmB+2WY+8IifRMMI\nG58+0dMi8hDwJ+Ao8CwlzzICeFBEFgDbgSvTEBoSbjb8tNNOA+J/vsOtBNizZ0/VcWi48zz11FOr\njqPi3hh16fI4z0rN7jwcrv69y/BJrKrfBr5dE3yYklcyjBOCoFcshIrb5mnx4sUAlMZPovPGG28A\n8P3vfx9obV8oDs6TuPOMu16v9JTfs57NnXcrqdw/wZ2Hw9W/L7Z2zjA8MU/kgbvDxmXIkCEpK8kW\np7doumvJSr95IsPw5IT3RG70ZurUqVXhW7duzV1DljryKDOOjlZq8MU8kWF4Yo3IMDw54R/n3ETp\n3Llzq8I7OzvrmbdUQ5Y68igzjo60NFRO8O7YsaNunO+r5OaJDMOTE94TuS2VfvzjH1eFu7uU+55n\nHhqy1JFHmXF0pKWhcsusrMowT2QYnpzwnsgRwiLQPDSEcN7QGh1ZlWGeyDA8Kbwnev/99wHYtGkT\nAKecckqf9u7DTi5d3PyjlNGMuBrq6YirIY8y89CQpAxfzBMZhicSdxl/FsyePVs7OjqOL5VfunQp\nEO4rAkb/wo3OLVmyBOh55UNENqrq7GbpzRMZhifWiAzDE2tEhuFJ0KNz7hOD5557blX4H/7wh162\nzWzSjg+tDPd5RUfUusvjvPK4flFtkmCeyDA8CdoTuY/eTpgwoW54Jc1s0o4PtYza8BDPK4/rF9Um\nCeaJDMMTmycyTnhsnsgwcsYakWF4Yo3IMDwJanRu2LBhAMybNw/oPfdhGFng5tTc/19czBMZhidB\neKL9+/ezZs2aRGndJ9bd5ulx3170TW+ki7sejsqPdGWF++zkww8/nCi9eSLD8CSIeSIRiSTCbUhe\nuUfZ2WefDcBHP/pRAG677TYAnn/++Uh5xE1vZMOgQYMA+PjHPw707AX34osvAvl8lgWweSLDaAVN\nG5GI3C0i+0RkU0XYWBH5rYhsLf8eUxF3q4hsE5GXReQfshJuGKEQZWDhHuAO4N6KsFuADaq6TERu\nKR9/Q0SmA1cBZwATgN+JyN+paiq9Q7dgcPr06cfD3OPY2LFjq44bPY7V5hE3vZENbmBny5YtAMyc\nORPouR4dHR0AdHV15aCub5p6IlV9HNhfE3wpsLr892rgsorwtap6WFVfAbYBZ6ek1TCCJOkQ93hV\ndTuO7wHGl/+eCDxVYberHNYLEVkILIxTaHd3d68wt03sd77zHQCeeeaZWHnETW9kg9viyg1pu+vk\nBhjOOeccANavXw/AsWPHWi2xId7zRKqqUUfXatKtBFZC9NE5wwiRpI1or4i0q2qniLQD+8rhu4FJ\nFXanlcMyo/bOFXey1De9kS6uL+Q80HnnnVcV/8QTTwDVG9XnTdIh7nXA/PLf84FHKsKvEpEhIjIF\nmAr8r59Ewwibpp5IRB4ALgTGicgu4NvAMuBBEVkAbAeuBFDVF0XkQeAl4CiwKK2RuUr27Nlz/O+b\nb74ZgFmzZgHwve99D4B77rknUh5J0xvpMmrUKAA+/elPAz2jqG77YOd53DbBIdG0Eanq1Q2i5jSw\nXwos9RFlGEUiiAWocVm7du3xvydOLA3+udGbqPM7Lo+k6Y10cX3S2vr/5S9/CbR2g/q42LIfw/Ck\nUAtQ6zF48GCgZwGje5Ev6nn5pjfSJY9XIfrAFqAaRisIxRN1Ae8ARdgjaxzh6yyCRghf5+mq+oFm\nRkE0IgAR6YjiOvOmCDqLoBGKo7MZ9jhnGJ5YIzIMT0JqRCvzFhCRIugsgkYojs4+CaZPZBhFJSRP\nZBiFJIhGJCJzy3sybCu/bp47IjJJRP5HRF4SkRdF5KZyeMP9JXLUOkBEnhWRXwWscbSIPCQiW0Rk\ns4h8MkSdSci9EYnIAGA58I/AdODq8l4NeXMU+JqqTgc+ASwq63L7S0wFNpSP8+YmYHPFcYgabwd+\nrarTgLMo6Q1RZ3xUNdcf4JPAbyqObwVuzVtXHZ2PABcDLwPt5bB24OWcdZ1G6R/wM8CvymGhaRwF\nvEK5D14RHpTOpD+5eyJKezDsrDhuuC9DXojIZGAm8DSN95fIix8CXwcqNx0ITeMUoAv4z/Jj509E\n5GTC05mIEBpR0IjICOAXwFdV9VBlnJZuobkNb4rI54B9qrqxkU3eGssMBGYBd6rqTEpLvKoe3QLR\nmYgQGlHL92WIiogMotSA7ldVt9v53vK+EtTsL5EH5wKfF5FXgbXAZ0RkDWFphNLTxS5Vfbp8/BCl\nRhWazkSE0IieAaaKyBQRGUxp88d1OWtCSrsJ3gVsVtUfVEQ12l+i5ajqrap6mqpOplRv/62q1xKQ\nRgBV3QPsFJGPlIPmUNpCICidicm7U1buVM4D/gL8H7Akbz1lTedRerx4Hniu/DMPaKPUkd8K/A4Y\nm7fWst4L6RlYCE4jMAPoKNfnfwFjQtSZ5MdWLBiGJyE8zhlGobFGZBieWCMyDE+sERmGJ9aIDMMT\na0SG4Yk1IsPwxBqRYXjy/ysREjSbVcfRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0fecef7490>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "from agentnet.experiments.openai_gym.wrappers import PreprocessImage\n",
    "#game maker consider https://gym.openai.com/envs\n",
    "def make_env():\n",
    "    env = gym.make(\"MsPacmanDeterministic-v0\")\n",
    "    return PreprocessImage(env,height=105,width=80,\n",
    "                           grayscale=True,\n",
    "                           crop=lambda img:img[:-25])\n",
    "\n",
    "#spawn game instance\n",
    "env = make_env()\n",
    "env.reset()\n",
    "observation_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "obs = env.step(0)[0]\n",
    "\n",
    "plt.imshow(obs[0],interpolation='none',cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into Qvalues using simple convolutional neural network.\n",
    "\n",
    "![scheme](https://s18.postimg.org/gbmsq6gmx/dqn_scheme.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import theano, lasagne\n",
    "from lasagne.layers import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#observation\n",
    "observation_layer = InputLayer((None,)+observation_shape,)\n",
    "\n",
    "#4-tick window over images\n",
    "from agentnet.memory import WindowAugmentation, LSTMCell\n",
    "\n",
    "prev_wnd = InputLayer((None,4)+observation_shape)\n",
    "new_wnd = WindowAugmentation(observation_layer,prev_wnd)\n",
    "        \n",
    "#reshape to (frame, h,w). If you don't use grayscale, 4 should become 12.\n",
    "wnd_reshape = reshape(new_wnd, (-1,4*observation_shape[0])+observation_shape[1:])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.nonlinearities import elu,tanh,softmax\n",
    "\n",
    "#network body\n",
    "conv0 = Conv2DLayer(wnd_reshape,32,3,stride=2,nonlinearity=elu)\n",
    "conv1 = Conv2DLayer(conv0,32,3,stride=2,nonlinearity=elu)\n",
    "conv2 = Conv2DLayer(conv1,64,3,stride=2,nonlinearity=elu)\n",
    "conv3 = Conv2DLayer(conv2,128,3,stride=2,nonlinearity=elu)\n",
    "        \n",
    "dense = DenseLayer(dropout(conv3,0.1),512,nonlinearity=tanh,name='dense')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "#baseline for all qvalues\n",
    "qvalues_layer = DenseLayer(dense,n_actions,nonlinearity=None,name='qval')\n",
    "        \n",
    "#sample actions proportionally to policy_layer\n",
    "from agentnet.resolver import EpsilonGreedyResolver\n",
    "action_layer = EpsilonGreedyResolver(qvalues_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.target_network import TargetNetwork\n",
    "targetnet = TargetNetwork(qvalues_layer,conv2)\n",
    "qvalues_old = targetnet.output_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=(qvalues_layer,qvalues_old),\n",
    "              agent_states={new_wnd:prev_wnd},\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[W, b, W, b, W, b, W, b, dense.W, dense.b, qval.W, qval.b]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params(action_layer,trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-22 03:16:52,267] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,299] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,326] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,350] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,374] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,398] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,424] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,448] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,472] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,496] Making new env: MsPacmanDeterministic-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,make_env, N_AGENTS) #may need to adjust\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "actions:\n",
      "[5 2 2 2 8 2 2 2 2 5]\n",
      "rewards\n",
      "[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "CPU times: user 228 ms, sys: 16 ms, total: 244 ms\n",
      "Wall time: 241 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(10)\n",
    "\n",
    "print('actions:')\n",
    "print(action_log[0])\n",
    "print(\"rewards\")\n",
    "print(reward_log[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards (Q-learning objective)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/py27/lib/python2.7/site-packages/agentnet/utils/logging.py:14: UserWarning: [Verbose>=1] optimize_experience_replay is deprecated and will be removed in 1.0.2. Use experience_replay parameter.\n",
      "  default_warn(\"[Verbose>=%s] %s\"%(verbosity_level,message),**kwargs)\n"
     ]
    }
   ],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "replay = pool.experience_replay\n",
    "\n",
    "_,_,_,_,(qvalues_seq,old_qvalues_seq) = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    optimize_experience_replay=True,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get reference Qvalues according to Qlearning algorithm\n",
    "from agentnet.learning import qlearning\n",
    "\n",
    "#crop rewards to [-1,+1] to avoid explosion.\n",
    "rewards = replay.rewards/10.\n",
    "\n",
    "#loss for Qlearning = \n",
    "#(Q(s,a) - (r+ gamma*r' + gamma^2*r'' + ...  +gamma^10*Q(s_{t+10},a_max)))^2\n",
    "elwise_mse_loss = qlearning.get_elementwise_objective(qvalues_seq,\n",
    "                                                      replay.actions[0],\n",
    "                                                      rewards,\n",
    "                                                      replay.is_alive,\n",
    "                                                      qvalues_target=old_qvalues_seq,\n",
    "                                                      gamma_or_gammas=0.99,\n",
    "                                                      n_steps=10)\n",
    "\n",
    "#mean over all batches and time ticks\n",
    "loss = elwise_mse_loss.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates\n",
    "updates = lasagne.updates.adam(loss,weights,learning_rate=1e-4)\n",
    "\n",
    "#compile train function\n",
    "train_step = theano.function([],loss,updates=updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-22 03:17:32,373] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:17:32,402] DEPRECATION WARNING: env.spec.timestep_limit has been deprecated. Replace your call to `env.spec.timestep_limit` with `env.spec.tags.get('wrapper_config.TimeLimit.max_episode_steps')`. This change was made 12/28/2016 and is included in version 0.7.0\n",
      "[2017-02-22 03:17:32,404] Clearing 2 monitor files from previous run (because force=True was provided)\n",
      "[2017-02-22 03:17:32,419] Starting new video recorder writing to /home/hedgedir/rl_projects/records/openaigym.video.0.34712.video000000.mp4\n",
      "[2017-02-22 03:17:34,054] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/hedgedir/rl_projects/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 333 timesteps with reward=190.0\n"
     ]
    }
   ],
   "source": [
    "action_layer.epsilon.set_value(0)\n",
    "\n",
    "untrained_reward = np.mean(pool.evaluate(save_path=\"./records\",\n",
    "                                         record_video=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./records/openaigym.video.0.34712.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./records/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {}\n",
    "loss,reward_per_tick,reward =0,0,0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAFkCAYAAAAQQyCBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xl8VOX1x/HPAQREZbOyuKMgggpKFFQWFVTctWKRoKJg\n625tWrdarQjW1hU3bN3FLWpBBTcQN36ICkoQEEFUNhVBkFVRtjy/P86kmQxJyDKTO8l836/XvCZz\n7zN3zlwjHJ7lPBZCQERERCRKtaIOQEREREQJiYiIiEROCYmIiIhETgmJiIiIRE4JiYiIiEROCYmI\niIhETgmJiIiIRE4JiYiIiEROCYmIiIhETgmJiIiIRK7cCYmZdTezMWb2nZnlm9kpxbRpZ2ajzWyV\nmf1kZpPNbNe48/XMbLiZLTeztWY20syaJVyjiZk9Y2arzWylmT1iZttV7GuKiIhIOqtID8l2wKfA\nJcAWG+GY2d7AROBzoAdwADAU+DWu2d3AiUCfWJudgVEJl3oWaAf0irXtATxYgXhFREQkzVllNtcz\ns3zgtBDCmLhjucCGEMK5JbynIbAM6BdCeCl2rC0wGzg0hDDFzNoBs4CsEMK0WJvewGvAriGEJRUO\nWkRERNJOUueQmJnhvRlfmtlYM1tqZh+Z2alxzbKAOsDbBQdCCF8Ai4DDYocOBVYWJCMxb+E9Ml2S\nGbOIiIhEr06Sr9cM2B64BvgbcDVwPPCimR0ZQpgItMB7UNYkvHdp7Byx5x/iT4YQNpvZirg2RZjZ\njkBvYAFFh4dERESkdPWBPYFxIYQfowgg2QlJQY/LyyGEe2M/zzCzw4GL8LklqdIbeCaF1xcREanp\nzsLncFa5ZCcky4FN+HyQeLOBrrGflwB1zaxhQi9J89i5gjaJq25qA03j2iRaAPD000/Trl27isYv\n5ZSTk8OwYcOiDiOj6J5XPd3zqqd7XrVmz57N2WefDbG/S6OQ1IQkhLDRzD4G2iac2gdYGPt5Kp60\n9ALiJ7XuDnwYa/Mh0NjMDoqbR9ILMGByCR//K0C7du3o1KlTEr6NlEWjRo10v6uY7nnV0z2verrn\nkYlsykO5E5JYLZDWeHIAsJeZdQRWhBC+AW4HnjOzicC7+BySk4AjAEIIa8zsUeAuM1sJrAXuBSaF\nEKbE2swxs3HAw2Z2MVAXuA/I1QobERGRmqciPSQH44lGiD3ujB0fAQwKIbxsZhcB1wH3AF8Ap4cQ\nPoy7Rg6wGRgJ1APGApcmfE5/4H58dU1+rO0VFYhXRERE0ly5E5IQwgS2slw4hPAE8EQp59cDl8ce\nJbVZBZxd3vhERESk+tFeNlIp2dnZUYeQcXTPq57uedXTPc88larUmk7MrBMwderUqZoIJSIiUg55\neXlkZWWBV0jPiyIG9ZCIiIhI5JSQiIiISOSUkIiIiEjklJCIiIhI5JSQiIiISOSUkIiIiEjklJCI\niIhI5JSQiIiISOSUkIiIiEjklJCIiIhI5Cqy26+IiIjEufVW+PRTWLcO/vMfaNky6oiqH/WQiIiI\nVMLixXDttTB3LowbB08/HXVE1ZMSEhERkUr4+GN/fvllOP54eOmlaOOprpSQiIiIVMKUKdCiBey6\nK/z2t/Dhh/D991FHVf0oIREREamEKVOgc2cwg5NOgtq1vbdEykcJiYiISAXl5/uQTefO/rppUzjq\nKA3bVIQSEhERkXIIAW66Ce6/H776ClavLkxIwIdt3n0XVq2KLsbqSAmJiIhIOfzznzB4MOTkwGOP\n+bGDDy48f8IJsGkTvPdeFNFVX0pIREREymjUKPjb3+C662Dvvb3+yD77QJMmhW323BP22gveeSey\nMKslJSQiIiJl9Oij0KMH3HwzPPCAHzvkkC3b9eoFb79dtbFVd0pIREREyiA/Hz76CI4+2lfU9OwJ\n99wDl1++ZduePeHzz7X8tzyUkIiIiJTBF1/AypVw2GGFx/74R+jSZcu2PXv6s4Ztyk4JiYiISBl8\n8AHUqlV0RU1JmjWDAw7QsE15KCEREREpgw8+8CSjYcOytS+YRxJCauOqKZSQiIiIlMEHH8Dhh5e9\nfc+esGgRzJuXuphqEiUkIiIiW7FiBcyZU76E5IgjvIy8hm3KptwJiZl1N7MxZvadmeWb2SmltP1P\nrM0fE47XM7PhZrbczNaa2Ugza5bQpomZPWNmq81spZk9YmbblTdeERGRyvroI3+On9C6NQ0b+pJg\nJSRlU5Eeku2AT4FLgBJHxszst0AX4LtiTt8NnAj0AXoAOwOjEto8C7QDesXa9gAerEC8IiIilTJ2\nLOy8sxc8K49evXylTX5+auKqScqdkIQQxoYQ/h5CGA1YcW3MbBfgHqA/sCnhXENgEJATQpgQQpgG\nDAS6mlnnWJt2QG/g/BDCJyGED4DLgX5m1qK8MYuIiFTU5s3wwgvQt6/XHymPnj1h+XL47LPUxFaT\nJH0OiZkZ8CRwWwhhdjFNsoA6wP86sUIIXwCLgILOsEOBlbFkpcBbeI9MMSu+RUREUmPCBFi6FPr1\nK/97Dz8c6tfXsE1ZpGJS67XAhhDC/SWcbxE7vybh+NLYuYI2P8SfDCFsBlbEtREREUm5556DVq3K\nVn8kUf360LWrEpKyqJPMi5lZFvBH4KBkXrc8cnJyaNSoUZFj2dnZZGdnRxSRiIhUB9dc43M+jj22\n8NiGDTByJFx0UfmHawp07w733+/1SCp6jWTKzc0lNze3yLHVq1dHFE2hpCYkQDdgJ+AbK7zrtYG7\nzOxPIYS9gCVAXTNrmNBL0jx2jthz4qqb2kDTuDbFGjZsGJ06dar0FxERkcwRAtx3H8yfXzQhmTDB\ny8WfeWbFr92xo88jWbwYdtml8rFWVnH/SM/LyyMrKyuiiFyyh2yeBDoAHeMei4Hb8EmqAFPxia69\nCt5kZm2B3YEPY4c+BBqbWXxPSy98Eu3kJMcsIiIZbulS+OUXmDSpaGXVTz+F7bf3Cq0VdeCB/jx9\neuVirOnK3UMSqwXSmsIVNnuZWUdgRQjhG2BlQvuNwJIQwpcAIYQ1ZvYo3muyElgL3AtMCiFMibWZ\nY2bjgIfN7GKgLnAfkBtCKLWHREREpLwKqqkuXgwLF8Kee/rrWbOgfXvfw6ai9tgDGjXyhOSEEyod\nao1VkVt8MDAN7+kIwJ1AHnBTCe2Lq1WSA7wKjATew3tR+iS06Q/MwVfXvAr8H3BhBeIVEREp1fz5\nhT9/8EHhz599BvvvX7lrm0GHDuoh2Zpy95CEECZQjkQmNm8k8dh6vK7I5aW8bxVwdnnjExERKa95\n82CnnaBpUx+26d/fi5nNnu0/V1bHjvDWW5W/Tk2mvWxERCTjzZ/vVVi7dvWEBGDBAli3Dvbbr/LX\nP/BAmDvXr5fo++9h0CDYcUf45pvKf1Z1pYREREQy3rx5Xmvk8MNh5kxYs6awumplh2zAe0jy8wuv\nmZ8P//439O7tidCYMZ6sPPts5T+rulJCIiIiGS++hyQ/3+eRzJrlk1F33rny199vP58YO306/PQT\nnH46XHqpzy+55Rb46is49dTMTkiSXYdERESkWtmwwYdKWrWCtm1h3329JkmTJp5IJKOY2bbb+rVv\nuQVuuAF+/tl7RU46qbBN//6elCRjIm11pB4SERHJaIsWee2Rvfby5GPwYHj9dXjtteQmBgMH+hLg\nAQPg44+LJiMAxx3nSVBCEdWtys+HvLzkxRkVJSQiIlLtbNwIX3+dnGsV1CDZK7Ym9He/80Joq1Yl\nZ0Jrgauugvfeg9tu816YRHXrwhln+LBNKK5gRjG++AKOOMKHmn74Yevt05kSEhERqVbWrYNTTvEh\nkIULK3+9efOgdm3YdVd/XasWDBniP3fsWPnrl8cZZ/jqnpkzt9527lxfvbNkCYwdC82abf096UwJ\niYiIVBvr13u10//7P08iXnyx8tecP9+HUurEzao89VT48EPo0aPy1y+PI46ABg18yGhrXnvNn6dN\n8/dVd0pIRESk2pgwwR9jxviS2VGjKn/NgiW/8czg0EOrfnfeevXg6KMLk43STJwIXbr4Xjs1gRIS\nERGpNqZP97+AjzoK+vTxImbffVe5a86dC61bJye+ZDjxRF92vHJlyW1CgPffh+7dqy6uVFNCIiIi\n1cb06T7htFYtn0dSpw689FLFr7dpk08MTebk1co64QRfOTNunL9evx4uvhief76wzRdfwLJlSkhE\nREQiMX164UTTJk18eOO//6349ebN87/w27dPTnzJsOuuvhnfs88WFkz7z3/gL3/xmingwzW1asFh\nh0UbazIpIRERkWph/XqYM6foypezzvIJrmVZlVKczz/353TqIQHf2+aVV6BNG08+7r/fh6YKekkm\nToSDDoIddog2zmRSQiIiItXC55/7EEt8QnLmmbD77nDrrRW75qxZ3tPSvHlyYkyWK67w6rGvvOJF\nzy69FI4/Hu64w+ePTJxY9SuAUk0JiYiIVAuffuqrXg44oPDYNtvAlVfCc8/58t3ymjUreeXhk23X\nXb2aa9u2/vrKK2HGDNhpJ69VcswxkYaXdEpIRESkWpg+Hfbee8tlruefD02beu9BeX3+eXrNHynN\nUUd5tdcLLoC33/Yek5pEm+uJiEi1ED+hNV6DBnDuuT4J9P77y97bsXmzz0kZNCi5caaKmZedr6nU\nQyIiImkvhJITEvBKpYsXl6+UfDqusMlkSkhERCTtzZzphcIOOaT484cf7s8TJ5b9mrNm+XO6rbDJ\nVEpIREQk7T3zjM8T6dmz+PNNm3pi8f77Zb/mrFnQuDG0aJGcGKVylJCIiEhay8+H3Fzo2xfq1i25\nXbdu5UtIPv3UV+yk4wqbTKSERERE0trEiV6T46yzSm/XrZuvmvnxx61fMwTfB6dr1+TEKJWnhERE\nRNLaM8/AHnsUzhMpSbdu/vzBB1u/5vz58P33SkjSiRISERFJW+vX+141/fv73i2l2WMP2GUXePBB\n71EpsHkzLF1atO2kSf68tSRHqo4SEhERSSt5efDOO/7z2LGwatXWh2vA54IMGeLJRqtWsM8+cOCB\n0KiRT1x9663Ctu+/78t9mzZNzXeQ8lNCIiIiaeXGG+H0032Z7zPPeO2Rsi7NHTTIe0eGD4fTTvPd\ncG+8EbKy4G9/87kj4ElLwRCPpAdVahURkbQydy6sXg033OCbyw0ZUr73b789XHhh0WOdOsHRR8Or\nr3oiMmsWXH118mKWylNCIiIiaWPjRq+guttu3sthBtnZlb9uz55ezfWqq3xPGFAPSbop95CNmXU3\nszFm9p2Z5ZvZKXHn6pjZrWY2w8x+irUZYWYtE65Rz8yGm9lyM1trZiPNrFlCmyZm9oyZrTazlWb2\niJltV/GvKiIi6W7BAti0Ce680+d+HHGE73pbWWZ+TbPCYaBWrSp/XUmeivSQbAd8CjwKvJhwrgFw\nIHATMANoAtwLjAY6x7W7Gzge6AOsAYYDo4DucW2eBZoDvYC6wBPAg8DZFYhZRESqgblz/fnQQ+H1\n15M76TQrC2bPTt71JLnKnZCEEMYCYwHMita3CyGsAXrHHzOzy4DJZrZrCOFbM2sIDAL6hRAmxNoM\nBGabWecQwhQzaxe7TlYIYVqszeXAa2Z2ZQhhSbm/qYiIpL25c2HbbX357m67RR2NVKWqWGXTGAjA\nqtjrLDwRerugQQjhC2ARcFjs0KHAyoJkJOat2HW6pDpgERGJxty50KbN1muOSM2T0v/kZlYP+Bfw\nbAjhp9jhFsCGWG9KvKWxcwVtfog/GULYDKyIayMiIjXMl196/RDJPClbZWNmdYD/4r0al6TqcxLl\n5OTQqFGjIseys7PJTsY0bRERSam5c+Gcc6KOombLzc0lNze3yLHVq1dHFE2hlCQkccnIbkDPuN4R\ngCVAXTNrmNBL0jx2rqBN4qqb2kDTuDbFGjZsGJ06darkNxARkaq2bp0XNVMPSWoV94/0vLw8srKy\nIorIJX3IJi4Z2QvoFUJYmdBkKrAJXz1T8J62wO7Ah7FDHwKNzeyguPf1AgyYnOyYRUQkel995c9t\n2kQbh0Sj3D0ksVogrfHkAGAvM+uIz+/4Hl++eyBwErCNmTWPtVsRQtgYQlhjZo8Cd5nZSmAtvjR4\nUghhCkAIYY6ZjQMeNrOL8WW/9wG5WmEjIlIzFSz5VQ9JZqrIkM3BwLv43JAA3Bk7PgKvP3Jy7Pin\nseMWe30U8H+xYznAZmAkUA9fRnxpwuf0B+7HV9fkx9peUYF4RUTSxs8/w08/QfPmW2+baWbMgCZN\nYMcdo45EolCROiQTKH2oZ6vDQCGE9cDlsUdJbVahImgiUkOsWuUbvc2Z46/z8uCgg0p/T00wcyZ8\n/z0cc4xXSS3J5s0wYgScemrp7aTm0kpvEZEUeeghKFjMcM89XhZ9xAho0ADeeivS0KrMOedA796+\nl0zBkExxXn0VFi2CSxP7yiVjKCEREUmBDRt8I7cBAzz5uPtu34F2wADvKZk4MeoIU2/aNJg+Ha69\nFhYu9O8eQuH5NWvgssu8t+j++6FLFzj44OjilWgpIRERSYEJE/wv3N12g+OPh19+Kdzuvnt3eP99\nyM+PNsZUe/xxaNEChg6Ff/8bJk8u2jP09tu+o+8hh/hx9Y5kNiUkIiIp8PLLsOeeMH68D9FceCHs\nvLOf694dVq6EWbMiDTGl1q/3XXXPOQfq1IFjj/XEY8iQwl6SWbOgcWO49VY48UT43e+ijVmipYRE\nRCTJQoDRo32C5t57w7x5cNddhecPPdT/kq7JwzajR8OKFXDeef7aDG64wXuG/i+23vLzz2G//eDK\nK30OSf36kYUraUAJiYhIkk2dCt99B6ed5q933BFq1y4836CBz5WoSQlJfr4Pwaxf7/vRXHKJ94q0\nb1/Y5qSTvNdo1Ch/PWuWJyQioIRERCTpRo+Gpk2hW7eS23Tv7j0F8ZM8q7Nnn4Wjj/YeoWOOgd/8\npnCFUQEz/96TJsGmTb4EWgmJFFBCIiKSZJ9+WjgsU5KePWHxYl+FAp6YlDU5WbAAHnww+mTmhRfg\n22/95wcfhM6d4aijfOjljTc8KUvUtat/5xkzfCWSEhIpoIRERCTJ5s+HVq1Kb9Orl1drHTHCX19+\nObRt64XEvv0Wzj/fJ8QW5/HH4aKL4IEHkht3efz8M2Rnw5lneszvvw9/+Qs89ZT3fJT0/bt29SJo\njz/ur5WQSAElJCIiSRRC2RKSbbaBs8+Gp5/2uRT//jcsX+41Stq3h8ceg+uuK/69s2dDrVqQkwMf\nfZT871AWs2b5vJEPPoCTT4ZmzQrnzJSmfXtfWfPUU96DohL6UkAJiYhIKULwf9GX1bJlsG7d1hMS\n8BUoy5f7X+gtW/put/37+1LZp56CTz7xR6I5c2DgQJ8YO3BgNEM306d7UnTRRV70bOBAqFt36++r\nVcuTrtWrPTlRmXgpoIRERKQUf/kL9OlT9vbz5/tzWRKS/ff3pGL+fLj+eu8xeOghLxaWne1F1R58\nsOh7Nm/2EuwdOnjBsTlzvJeiqs2Y4bvy3nmnx/7nP5f9vV27+rOGaySeEhIRkVLMnQvjxvkEzLIo\nT0ICPuxy6KEwaFDR47VrwwUX+OqVRx7xv/R/+sl7I9avh3339Qmke+xROB+jKk2fDh07+hLmoUN9\nyKaslJBIcZSQiIiUYsUK+PXX4odOijN/vs+RaNy4bO3794cPPyx+uOP8871H5IIL4B//8OqvBbsF\n77uvD3+cdx48/7xPMq0qIXgPSYcOFXv/oYf6MNVxxyU3LqnelJCIiJRi5Up/LmsRs7JMaC2rli29\nyNiqVd4bMX68JyTbbQe77uptzj3Xe05GjkzOZ5bFokU+B6Rjx4q9v359GDMG2rRJblxSvSkhEREp\nRUFCUlDufGuSmZCAzyNp2NCLjY0f7yts2rb13hHwzzriCK8JUlUKaqdUtIdEpDhKSEREShCCJyQt\nW3p10bKstkl2QlLgmGPg++/hlVd8uCbe0UeXPb5kmDEDmjQp7KURSQYlJCIiJVi3zieznnKKD1HM\nnFl6+82bfTgjFQlJ9+5Qrx4sXbplQtKtm8f32WfJ/9wCS5d6ddnWrX1JcocOWrIryaWERESkBAXD\nNb17+6TTrc0jWbwYNm5MTUKy7baelMCWCUmXLl5oLVWb9c2eDYcc4s+HHw7ffFP6Pj0iFaGERESk\nBAUJyc47+1/IW5tHUt4lv+V1zDH+nJiQbLtt6nYPDgH+8Adf3vvxx/Dkk77y6Kabkv9ZktlK2fpJ\nRCSzrVjhz02aQI8eXs49hJKHKgoSkj33TE0855wD330H7dptea57dy9DX1p8FfHWWz4/5bXXCueM\n1K+fvOuLFFAPiYhICQp6SJo08b/wly71Zbgl+eQT2H1377FIhZYt4Z57it9FuHt3HzIqSIoqY+lS\nL9Q2fjzceKMPCR1/fOWvK1Ia9ZCIiJSgICFp3NjnTtSq5cMi++yzZdv1672q6u9/X7UxFjj8cH+e\nOBH22qty13rqKa/+WlABduxYTWCV1FMPiYhICVasgB128AmjjRrBgQeWPI/klVe8/cCBVRtjgaZN\nvQLqPffApk2Vu9bLL/vKotdfh9tvh2OPTU6MIqVRQiIiUoKVK324pkD37iVPHH38cU8IEiecVqV7\n7/WiZXfdVfFrLF3qm/X99rc+THPlleodkaqhhEREpASJCUmPHj5H49tvi7abNcuHNaLqHSlwyCHw\npz/5vI+vv67YNV55xROQk05KbmwiW6OERESkBCtX+lBIgYLaGyec4CtpDjgAsrJg//2heXPo2zeS\nMIsYMsT3unnssYq9/+WXvSfoN79JblwiW1PuhMTMupvZGDP7zszyzeyUYtoMMbPFZrbOzMabWeuE\n8/XMbLiZLTeztWY20syaJbRpYmbPmNlqM1tpZo+Y2Xbl/4oiIhWzYkXRHpJmzeDii31TuH79fA+Z\n9u3hmWfgq6/KvsNvKm23nRdye+MNf71pk/febNzoS4LvuQeGDi3+vT/95Mt8Tz216uIVKVCRVTbb\nAZ8CjwIvJp40s2uAy4ABwALgZmCcmbULIWyINbsbOB7oA6wBhgOjgO5xl3oWaA70AuoCTwAPAmdX\nIGYRkXJbuXLL/VoeeCCaWMrjuON8xc+SJTBqFFx2mdcuadPGd9mtUwcuvbRo7w9478j69XD66dHE\nLZmt3D0kIYSxIYS/hxBGA8VNdboCGBpCeDWE8BmemOwMnAZgZg2BQUBOCGFCCGEaMBDoamadY23a\nAb2B80MIn4QQPgAuB/qZWYvyf00RkfJLHLKpLnr39uc33oD77oMjj/QhmHfegeHDfc+dV17xHpO+\nff0YeE9Pt26wxx6RhS4ZLKl1SMysFdACeLvgWAhhjZlNBg4DXgAOjn1ufJsvzGxRrM0U4FBgZSxZ\nKfAWEIAuwOhkxi0iUpzEIZvqolkzLyV/002wcCE89JDPC/n1Vy/a9uyz3nOyyy7w3//68t4ePbwQ\n2v33Rx29ZKpkT2ptgScNSxOOL42dAx+G2RBCWFNKmxbAD/EnQwibgRVxbUREUiYEWLWqeiYk4Et2\nFy6Ejh09GTErrCDbpw+8+SbccINPzN1mG29vBmecEW3ckrm0ykZEpBhr1/rQRnVNSE480Z8vv3zL\nOiKnn+5zRT76yJcIX3ed75Fz3HFaXSPRSXbp+CX4vJLmFO0laQ5Mi2tT18waJvSSNI+dK2iTuOqm\nNtA0rk2xcnJyaNSoUZFj2dnZZGdnl++biEhGK9hYrzrOIQHo3BnefttXAiXaYw8f0lmzBk47zVfg\nvPGGJy9S8+Xm5pKbm1vk2OrVqyOKppCFECr+ZrN84LQQwpi4Y4uB20MIw2KvG+LJyYAQwn9jr5cB\n/UIIL8XatAVmA4eGEKaY2b7ALODggnkkZnYs8Dqwawhhi6TEzDoBU6dOnUqnTp0q/J1ERACmTYNO\nneDjj/0v75rmyy+956R16623lZovLy+PrKwsgKwQQl4UMZS7hyRWC6Q1hSts9jKzjsCKEMI3+JLe\n683sK3zZ71DgW2ITUWOTXB8F7jKzlcBa4F5gUghhSqzNHDMbBzxsZhfjy37vA3KLS0ZERJItfqff\nmqhNm6gjECmqIkM2BwPv4pNXA3Bn7PgIYFAI4TYza4DXDGkMTASOj6tBApADbAZGAvWAscClCZ/T\nH7gfX12TH2t7RQXiFREpt+o+ZCNS3ZQ7IQkhTGArk2FDCIOBwaWcX4/XFSlxxDKEsAoVQRORiKxc\n6UMaCVPSRCRFtMpGRKQYK1d6MlJLf0qKVAn9ryYiUoxPPoG99446CpHMoYRERCTBTz/Bq6/C734X\ndSQimUMJiYhIgldegV9+gTPPjDoSkcyhhEREJMFzz0GXLrDnnlFHIpI5lJCIiMRZtQrGjoV+/aKO\nRCSzKCEREYlZvBj69/c9bDR/RKRqJXsvGxGRamnxYth/f6hXD15+GXbZJeqIRDKLEhIREWDKFK89\nMn++5o6IREFDNiIiwMKFUL++74QrIlVPCYmICLBggScjZlttKiIpoIRERATvIVHviEh0lJCIiKCE\nRCRqSkhERPAhG01mFYmOEhIRyXg//QQrVqiHRCRKSkhEJOMtXOjP6iERiY4SEhHJeAsW+LN6SESi\no4RERDLewoVQpw60bBl1JCKZSwmJiGS8hQth992hdu2oIxHJXEpIRCTjFRRFE5HoKCERkYynGiQi\n0VNCIiIZZ+FCmDu36GutsBGJlnb7FZGM89vfwrRp0Lq194wsWaIeEpGoKSERkYyyaRPMmgXnnQcN\nGsDy5dCvHxx9dNSRiWQ2JSQiklEWLIANG+Css5SEiKQTzSERkYwye7Y/t2sXbRwiUpQSEhHJKHPm\nwA47wM47Rx2JiMRTQiIiGWX2bNh3XzCLOhIRiZf0hMTMapnZUDObZ2brzOwrM7u+mHZDzGxxrM14\nM2udcL6emQ03s+VmttbMRppZs2THKyKZpSAhEZH0kooekmuBC4FLgH2Bq4GrzeyyggZmdg1wGXAB\n0Bn4GRhnZnXjrnM3cCLQB+gB7AyMSkG8IpIhQvAhG80fEUk/qUhIDgNGhxDGhhAWhRBeBN7EE48C\nVwBDQwiajS8wAAAgAElEQVSvhhA+AwbgCcdpAGbWEBgE5IQQJoQQpgEDga5mFn8dEZEtfP019O4N\nP/5Y9PjSpbBqlXpIRNJRKhKSD4BeZtYGwMw6Al2B12OvWwEtgLcL3hBCWANMxpMZgIPxJcnxbb4A\nFsW1EREp1i23wJtvQm5u0eNz5vizEhKR9JOKhORfwPPAHDPbAEwF7g4hPBc73wIIwNKE9y2NnQNo\nDmyIJSoltRER2cK338JTT3nRs6efLnpu9myoU8crtIpIeklFYbQzgf5AP+Bz4EDgHjNbHEJ4KgWf\nV0ROTg6NGjUqciw7O5vs7OxUf7SIpIE774Ttt4c77oDzz4cvv4Q2bfzcnDmw996wzTbRxigSpdzc\nXHITug9Xr14dUTSFLISQ3AuaLQL+GUL4d9yxvwFnhRDax4ZsvgYODCHMiGvzHjAthJBjZkcBbwFN\n4ntJzGwBMCyEcE8xn9sJmDp16lQ6deqU1O8kItXD6tVeX+Qvf4G//hVatICcHBg82Ce07r8/dOrk\nPSgiUigvL4+srCyArBBCXhQxpGLIpgGwOeFYfsFnhRDmA0uAXgUnY5NYu+DzT8CHeTYltGkL7A58\nmIKYRaQGePNNWLcOfv972HZbOOMMTz42b/bN9D7/HPr3jzpKESlOKoZsXgGuN7NvgVlAJyAHeCSu\nzd2xNl8BC4ChwLfAaPBJrmb2KHCXma0E1gL3ApNCCFNSELOIVDMLF8I330DnzlA3VjDg9de9F2T3\n3f31hRfCY4/BiBHw2WfQrBkcc0x0MYtIyVKRkFyGJxjDgWbAYuDfsWMAhBBuM7MGwINAY2AicHwI\nYUPcdXLwnpaRQD1gLHBpCuIVkWomPx9OOsmTjAYN4K674A9/8ITkvPMK23XuDNnZcN11/jo72ye1\nikj6Sfr/miGEn4E/xx6ltRsMDC7l/Hrg8thDROR/XnzRk5ERIzwJufJK2GUX+OEHOPHEom3/9S9o\n2xZ+/RXOOSeaeEVk67SXjYhUC9OnQ9++nogMHQpHHw0DBsADD3ivxznnQKNGcFhCpaLdd4ebboIj\njvAJrSKSnpSQiEi18Npr8N//QseOMGMG3HijH2/aFG64wSuw9u5d/JLeq6+G997Thnoi6UyjqSJS\nLSxaBPvtB6ed5slHt26F5y69FN54AwYOjC4+EakcJSQiUi0sXOgFzm6+ectz9erB+PFVH5OIJI+G\nbESkWli0qHA5r4jUPEpIRCTthaCERKSmU0IiImlv1Sr46SclJCI1mRISEUl7Cxf68x57RBuHiKSO\nEhIRSXuLFvmzekhEai4lJCKS9hYt8v1qmjWLOhIRSRUlJCKS9hYtgt12g1r6E0ukxtL/3iKSltav\n94Jn33/vc0g0f0SkZlNhNBFJS5Mn+z41jRp5D8m++0YdkYikknpIRCQtffyxPz/2GMybpwmtIjWd\nekhEJC19/DHssgt8952/VkIiUrOph0RE0tLHH8PvfgeHHeavNYdEpGZTQiIiaWf5ch+mOeQQuOgi\nP7bXXtHGJCKppSEbEUk7n3ziz4ccAnvvDW3bKiERqenUQyIiKfXJJ76Etyy++QZWr/bhmsaNoXVr\nrz3SpUtqYxSR6CkhEZGUeeQR7+W46qrCYyHArbfCccfBhg2Fx9etg4MPhgMPhNGj/Wezqo9ZRKKh\nhEREUuKll+DCC3245d//hq++grVr4Ywz4NprYdw4ePzxwvYjRvjckQYNYOpUT2REJHMoIRGRpAsB\nLr4YTjnFh2yaN4dLLvGhl/HjPVnJzoabb/bhnM2b4c474fTT4aOP4K9/hfPPj/pbiEhV0qRWEUm6\n2bNh6VIv/b799jB0KAwa5NVWp0zx5333hf32g1tugR13hK+/hmefhR128GMiklmUkIhI0r33Hmyz\nTWENkQEDYLvtfN5Iw4Z+bN99YeBAGDLEXx91FHTuHEm4IpIGlJCISNK9954nF9tt569r14a+fbds\n99BDkJMD+fnQqlWVhigiaUZzSEQkKf75Tzj5ZJ8P8t57cOSRW39PrVo+bHPAAT60IyKZSz0kIlJp\n334LN93kE1SvvBKWLStbQiIiUkAJiYhU2j/+4T0cJ50Ed99ddP6IiEhZpGTIxsx2NrOnzGy5ma0z\ns+lm1imhzRAzWxw7P97MWiecr2dmw2PXWGtmI82sWSriFZGK+e47L2L2yCNwzTUwfDg0auQ1RArm\nj4iIlEXSe0jMrDEwCXgb6A0sB9oAK+PaXANcBgwAFgA3A+PMrF0IoaB2493A8UAfYA0wHBgFdE92\nzCJSPpMnw+DBMHasv27b1pf4NmgAY8b4s4hIeaRiyOZaYFEI4fdxxxYmtLkCGBpCeBXAzAYAS4HT\ngBfMrCEwCOgXQpgQazMQmG1mnUMIU1IQt4iUYN062HZbL+X+7rvQqxe0awdPPgldu8Iee/hKGoAe\nPaKNVUSqp1QM2ZwMfGJmL5jZUjPLM7P/JSdm1gpogfegABBCWANMBgpGnQ/Gk6X4Nl8Ai+LaiEgV\nWL/eV8Kceir8/LNXYO3WDWbMgHPO8V14C5IREZGKSkUPyV7AxcCdwD+AzsC9ZrY+hPAUnowEvEck\n3tLYOYDmwIZYolJSGxGpAs88AwsW+EqaDh1g0SIYNUpJiIgkVyoSklrAlBDCDbHX081sf+Ai4KkU\nfF4ROTk5NGrUqMix7OxssrOzU/3RIjVOfj7ccYfvSZOdDf37w9VXe4+JiFRPubm55ObmFjm2evXq\niKIplIqE5HtgdsKx2cDpsZ+XAIb3gsT3kjQHpsW1qWtmDRN6SZrHzpVo2LBhdOrUqbQmIlJGY8f6\nvjQPPeTDNJ07w557Rh2ViFRGcf9Iz8vLIysrK6KIXCrmkEwC2iYca0tsYmsIYT6eVPQqOBmbxNoF\n+CB2aCqwKaFNW2B34MMUxCwiCfLzfVO8Ll184ir4fJFaqu8sIimQih6SYcAkM/sr8AKeaPwe+ENc\nm7uB683sK3zZ71DgW2A0+CRXM3sUuMvMVgJrgXuBSVphI1I1HnkEPvoIJkzw1TUiIqmU9IQkhPCJ\nmf0W+BdwAzAfuCKE8Fxcm9vMrAHwINAYmAgcH1eDBCAH2AyMBOoBY4FLkx2viGxpyRIvdDZokJbx\nikjVSEnp+BDC68DrW2kzGBhcyvn1wOWxh4ikUAgwbhx8+il8/LHXGqlTB267LerIRCRTaC8bEWH4\ncLj8cthhB+jYEf74R19Vs+OOUUcmIplCCYlIhtuwAW69Fc46C556SvNFRCQami8vkuGeftqLnl13\nnZIREYmOEhKRDLZsGfzrX3D66dC+fdTRiEgmU0IikoE2bYIzz4SWLeGbb+D666OOSEQynRISkQz0\n3nvwwgvwz3/63jQHHRR1RCKS6TSpVSQDPf+8V1298krNGxGR9KAeEpE08+GHsHFj6q6/YQO8+KIP\n2SgZEZF0oR4SkTSyaBEcfjj85S++y26B/HxYtw62377i1x4xwq/TvDmsWOEJiYhIulBCIpJGpk71\n57vu8pUvhx/uSUrfvr40d+5caNCg/Nf94Qe46CL49Vdo0gTatoUOHZIbu4hIZWjIRiSNTJsGO+3k\nO+z27++9GAcdBIsX+/4yDz1Usevef7/v0vvoo95LMmiQhmtEJL0oIRFJI9OmeQIyYgS0agXLl0O/\nfn58wACvqPrLL+W75s8/e2n4P/zBE5EffoCrrkpN/CIiFaWERKQEn33mCUFVKkhI9tnHN7h7+21P\nJnbc0Sup/vADPPJIYfvNm0u+1iefwDnnwKmnwurVkJPjx+vWVe+IiKQfJSQiJTjtNB82SYW1a2Hg\nQPjTnwqPLVsG331Xck2Q1q29l+Rvf4MZM2DWLNhtN+jTB9as2bL9tdfC+PGwfj38/e+wxx6p+S4i\nIsmgSa0ixfj1V5g3D77+Gt55B3r2TM511671no9rroGvvvKKqWecAd26ee8IlF6k7L77YPp0OOEE\nX767447w1ltwyCGefOy+u7f7/HPvXXnmmdQlVSIiyaQeEpFifP01hOArUq691n8GnxA6dWrh6/LI\nzfUE4tRToX597+XIyoI//9mvO22aL+tt3brka2y/Pbz6qk9Q3XVXeP99H5r55RfvcSmI6777oEUL\nT3ZERKoDJSQixfjyS39+4AH4+GN47jl/PWwYHHywJxHlSUrmzvVJpaed5tfOy4N27fx6H3/sk1U/\n+QQ6dvRkozQ77+w9IB995AlOmzY+r+Sdd/z5m2/gySfhwgt9voiISHWgIRuRYsydCzvs4MtuR42C\nyy7zv/gHD/aE5O67fSlu9+6+iuWDD7zg2A03+LyOeOvX+7DJzjvDY48VLW7WvbvPI7nuOn99+eVl\niy+xQNqxx/oKmksu8WGghg09IRERqS7UQyJSjLlzfaWLGfznP1CvHnTtCttu63M1Hn/cS7z/+c8w\ndKjPDXn5ZX/PtdfCqlV+nRA8MZg503tZiqu0OmyY946cdZY/Kuquu3wlzVNPeS9My5YVv5aISFVT\nD4lIMb780pML8GGRJ57wiaR33AGNG8N55/kjBH/UquVJye23w513wsMP+5LbELymyLPPQqdOJX9e\nVhY8/XTlYm7UCG67rXLXEBGJihISkWLMnQtHHln4+thjvSZJ48ZF25kV1vTYYQcYMgQuvhj+9S8f\n6vn2Wx/myc6uqshFRKonJSQiCdas8TLtBT0kBRKTkZK0bAn33OPzTBYtKlyKKyIiJdMcEslYEyf6\nRNRzz4XJkwuPf/WVPycmJOVl5sXIVBVVRGTrlJBIRgrBi5PtsANMmuSFyQp22p0715/btIkuPhGR\nTKOERDLS2LG+Smb4cK/p0aGDT0L95RdPSHbaqexDNCIiUnmaQyIZJwTf26VrV5+sauaFxLKy4Ljj\nfCJqZYdrRESkfNRDIkm1cqUXBnvttagjKdlnn3ndj+uuK5zfsd9+XuV040avoPrHP0Ybo4hIplEP\niSTV8897D8M113hvQ+3aUUe0pZkz/fnww4seP/tsf4iISNVLeQ+JmV1rZvlmdlfC8SFmttjM1pnZ\neDNrnXC+npkNN7PlZrbWzEaaWbNUxyuV8+STPtwxa5YnJ+lo5kzfmE5zRERE0kdKExIzOwS4AJie\ncPwa4LLYuc7Az8A4M4vfCuxu4ESgD9AD2BkYlcp4pXK+/NInig4ZAief7Pu6DBvmSUp+ftTRFZo5\nEw44IOooREQkXsoSEjPbHnga+D2wKuH0FcDQEMKrIYTPgAF4wnFa7L0NgUFATghhQghhGjAQ6Gpm\nnVMVs1TOU0/5pm6nnAI33ww//uiTR88916uXpktS8tlnSkhERNJNKntIhgOvhBDeiT9oZq2AFsDb\nBcdCCGuAycBhsUMH4/Nb4tt8ASyKayNp5Ndffb+X3/3ON6Dr0ME3mFu71jeie/hhnygaQrRxrlkD\nCxfC/vtHG4eIiBSVkkmtZtYPOBBPLBK1AAKwNOH40tg5gObAhliiUlIbSSN33AHff++73yY67zxf\nvXLBBZ4IXHRRlYf3P7Nm+bN6SERE0kvSExIz2xWf/3F0CGFjsq+/NTk5OTRq1KjIsezsbLK1u1nK\nLFgAt9wCOTnQvn3xbf7wB5g+3XtJGjWCpk3hkEP8OZnWrYMZM+Dgg6FO3G/38uX+uTNn+sqfdu2S\n+7kiItVFbm4uubm5RY6tXr06omgKWUhyH7qZnQq8CGwGCnbxqI33imwG9gW+Ag4MIcyIe997wLQQ\nQo6ZHQW8BTSJ7yUxswXAsBDCPcV8bidg6tSpU+lU2j7vknR9+3r59TlzvBR7STZsgF694P33/fU+\n+8CUKZ4oAKxY4a+POabiy4UHD4abboIdd/Tk5+9/h2XLPFHq3BlatYJ33vHqrCIi4vLy8sjKygLI\nCiHkRRFDKoZs3gISO8SfAGYD/wohzDOzJUAvYAb8bxJrF3zeCcBUYFOszUuxNm2B3YEPUxCzVNDC\nhTBqlJdgLy0ZAahbF959F+bN8/klxx4LAwZ478mDD3o5902b4JJL4P77K7Yp3csv+3X32QduvNGX\n9777rs9xef11qF/fVwCJiEh6SXpCEkL4GSjy708z+xn4MYQwO3bobuB6M/sKWAAMBb4FRseuscbM\nHgXuMrOVwFrgXmBSCGFKsmOWinvoIdh++7IXFKtTp7As+9NPe3IwZowPsQwb5nNN/vxn2H13L65W\nHgsX+rDQ8897r82vv/p8lY0bfWLtrFk+10XzR0RE0k9VVWotMi4UQrjNzBoADwKNgYnA8SGEDXHN\ncvAhnpFAPWAscGnVhCtlsX69l1s/91xPSsrrpJPgjTd8HknnuMXcK1bAtdf6XjPdupX9emPGwDbb\nQO/e/vq++3w+SdOmHuPGjR7zGWeUP1YREUmtpM8hiYrmkFS93Fzo39/nYyRzkujmzZ6MrF0L06b5\nUE9ZHHOMD/O8+WbRa0F6lrAXEUkX6TCHRJvrSYU9+ST06JH8FSu1a/tQ0BdfwO23l9522TK4+mq4\n80547z049dQtr6VkREQk/WlzPamQX37xBOAf/0jN9Tt0gCuv9Impe+0Fxa3aXrIEjj4aFi3yFTxm\nXiVWRESqHyUkUiETJvik0eOOS91n3HyzJx1nnQUvvQRffw1NmsDpp8PSpV4ZdtMmXyrcqpXPPWnZ\nMnXxiIhI6ighkQp54w1fCZPKAmN16sBjj3mSMX48HHggfPON1xfZYQc48UTfyG+vvby9khERkepL\nCYlUyNix3jtSkVoh5VGrFvzzn/4osGaN75ezzTap/WwREak6mtQq5TZvHsydC8cfH83nN2yoZERE\npKZRQiLlNnq0D6f07Bl1JCIiUlMoIZFy+fFHX1nTr5/3VIiIiCSDEhIp1ZdfevGz7bbzjequuspX\nttxxR9SRiYhITaJJrVKiRYugY0cvvX7WWXDLLV759IEHoHnzqKMTEZGaRAmJlOjee6FePS8N37Ah\n/P73Xpb9gguijkxERGoaJSRSrLVr4eGH4eKLC+eKdO5cdBM8ERGRZNEcEinWY4/BunVw2WVRRyIi\nIplACYlsYeNGuOce6NsXdt016mhERCQTaMgmQ4Xg80FGjPCN6Z5+GurX93NPPQXz5/v+MSIiIlVB\nPSQZ6r//9dLvn34Kr70G550H+fneO3LzzdCnj6+wERERqQrqIclQEyZA27Ywa5b3hJxxhicju+zi\nvSOjR0cdoYiIZBL1kGSoadOgUyffHO/00+HRR2H2bLjvPjjzTDjggKgjFBGRTKKEJANt3gzTp8NB\nBxUeGzjQ640sWQJPPBFZaCIikqE0ZJOBvvzSl/R26rTlOVVgFRGRKKiHJANNm+bP8T0kIiIiUVJC\nUsPk53sNkeHDS24zbRrsvrvvUSMiIpIONGRTgyxbBmef7fVFABYv9iW8ZkXbTZum3hEREUkv6iGp\nQa66Cj75xBOSO+7w3XlPPRUmTfJCaODPeXlKSEREJL2oh6SG+OknGDkSrr0WjjnGH7vsAoMHQ7du\nvkFeu3aw116wYoUSEhERSS9KSGqIUaPg55/hnHMKj/Xr5/vRvPOO95zMng1z5kCbNnD44dHFKiIi\nkkgJSQ0xYgQcdRTssUfR47VqwdFH+0NERCRdJX0OiZn91cymmNkaM1tqZi+Z2T7FtBtiZovNbJ2Z\njTez1gnn65nZcDNbbmZrzWykmTVLdrw1wcKF8O67cO65UUciIiJSMamY1NoduA/oAhwNbAO8aWbb\nFjQws2uAy4ALgM7Az8A4M6sbd527gROBPkAPYGdgVArirfZGjvSdevv0iToSERGRikn6kE0I4YT4\n12Z2HvADkAW8Hzt8BTA0hPBqrM0AYClwGvCCmTUEBgH9QggTYm0GArPNrHMIYUqy467O3nwTjjgC\ntt8+6khEREQqpiqW/TYGArACwMxaAS2AtwsahBDWAJOBw2KHDsaTpfg2XwCL4toI8OuvMHGir6oR\nERGprlKakJiZ4UMv74cQPo8dboEnKEsTmi+NnQNoDmyIJSoltRHggw/gl180aVVERKq3VK+yeQBo\nD3RN8edkrPHjoVkzOOCAqCMRERGpuJQlJGZ2P3AC0D2E8H3cqSWA4b0g8b0kzYFpcW3qmlnDhF6S\n5rFzJcrJyaFRo0ZFjmVnZ5OdnV2h75Huxo/33pFaqrkrIiJlkJubS25ubpFjq1evjiiaQhYKaoon\n86KejJwKHBFCmFfM+cXA7SGEYbHXDfHkZEAI4b+x18vwSa0vxdq0BWYDhxY3qdXMOgFTp06dSqdO\nnZL+ndLRjz/CTjvBY4/BeedFHY2IiFRXeXl5ZGVlAWSFEPKiiCEVdUgeAM4C+gM/m1nz2KN+XLO7\ngevN7GQzOwB4EvgWGA3/m+T6KHCXmR1pZlnAY8Ckmr7C5uWX4cILYd4WaZxbscIrsq5a5ct869eH\n3r2rNkYREZFkS0VH/0VAQ+A9YHHco29BgxDCbXitkgfx1TXbAseHEDbEXScHeBUYGXetGltpY+VK\nGDQIfvtbePZZ33fm2mthTdyA1TvvQMuW0KQJ7LsvzJjhQzYtW0YXt4iISDKkog5JmZKcEMJgYHAp\n59cDl8ceNVYI8J//wPXXw8aNPvzSty/cfjvcdhs88QRcdplvinfRRXDkkXDyyTBzJlxxBbRvH/U3\nEBERqTztZROxV1+FSy6BgQPhllugRWxR8+DBcP758Le/eWKydi106QIvvgjbbRdpyCIiIkmnhCRC\nIcDf/+5VVh99FMyKnt9tN3jySdi8Gb7+GvbcE+rWLfZSIiIi1ZoSkirw4IPwxhtw+eWwyy4+76N1\na1i3Dj79FN57b8tkJF7t2rDPFtsTioiI1BxKSFLss8/gj3+ERo1g9Gg/VqcObNrktUN69fIeEhER\nkUymhCSFNm70+iCtW8PUqfDRR94rcuSRkJcHjz8Of/pT1FGKiIhETwlJEi1bBhdfDIcfDh07ws03\n+5DMhx96vZAjjyxs262bP0REREQJSVJddx28/jq88gps2AD77ec/H3JI1JGJiIikNyUkSfLxx75S\n5r77IDvb54507eoTUkVERKR0SkiSIARfQdOhg5d9r1MHevSIOioREZHqQwlJEowdC5Mnw9tvezIi\nIiIi5aNN65Pg1luhc2c46qioIxEREame9O/5Spo8GSZMgFGjSi9uJiIiIiVTQlJOGzfCc8/B3XfD\nqlV+bJ994NRTo41LRESkOlNCUowPPvAN7PbZx1fLfP017L8/fPONFzKbOxeOP94rrM6c6bvxajWN\niIhIxSkhibNpE+TkwP33l9zmyCPh+efhwAOrLCwREZEaTwlJzNq10K8fjBvnCUnHjvDFF9C+PbRp\nA7Nm+XBNr16aKyIiIpJsGZ2QLF4Mc+ZA3bo+7DJ/vldaPfZYPx9f2l0b4ImIiKRORiUkq1Z5NdW6\ndWHGDBgxwns9APbYAyZN8rkiIiIiUrUyKiF5+GG45hqoVw+aNIF//MNXx6xd6xNYd9gh6ghFREQy\nU0YlJGPGwMknw+jRUUciIiIi8TKmUuvy5b6c95RToo5EREREEmVMQvL665CfDyeeGHUkIiIikihj\nEpIxY6BLF2jRIupIREREJFFGJCTr13t9kZNPjjoSERERKU6NT0h++cULnv3yC/TpE3U0IiIiUpwa\nvcpm0ybo3Rs++cRX1uy7b9QRiYiISHFqdEIydixMnAhvveUl30VERCQ91eghm8cfhw4doGfPqCOp\nuXJzc6MOIePonlc93fOqp3ueedI+ITGzS81svpn9YmYfmdkhZXnf8uXwyiswcKA2w0sl/aFR9XTP\nq57uedXTPc88aZ2QmNmZwJ3AjcBBwHRgnJn9ZmvvffZZCAHOOivFQYqIiEilpXVCAuQAD4YQngwh\nzAEuAtYBg0p6w7Jl8J//wJ13+jLfnXaqqlBFRESkotJ2UquZbQNkAbcUHAshBDN7CzispPddfjnM\nmwc9esDQoVUQqIiIiFRa2iYkwG+A2sDShONLgbbFtK8PMGjQbLp0gUaNvCBaXl6Ko8xwq1evJk83\nuUrpnlc93fOqp3tetWbPnl3wY/2oYrAQQlSfXSozawl8BxwWQpgcd/xWoEcI4bCE9v2BZ6o2ShER\nkRrlrBDCs1F8cDr3kCwHNgPNE443B5YU034ccBawAPg1pZGJiIjULPWBPfG/SyORtj0kAGb2ETA5\nhHBF7LUBi4B7Qwi3RxqciIiIJE0695AA3AU8YWZTgSn4qpsGwBNRBiUiIiLJldYJSQjhhVjNkSH4\nUM2nQO8QwrJoIxMREZFkSushGxEREckM6V4YTURERDKAEhIRERGJXI1ISCq6AV+mM7MbzSw/4fF5\nQpshZrbYzNaZ2Xgza51wvp6ZDTez5Wa21sxGmlmzhDZNzOwZM1ttZivN7BEz264qvmPUzKy7mY0x\ns+9i9/eUYtpUyT02s93M7DUz+9nMlpjZbWZWI/4MiLe1e25mjxfze/96Qhvd83Iws7+a2RQzW2Nm\nS83sJTPbp5h2+l1PkrLc8+r2u17t/wNZJTbgEwA+wycMt4g9uhWcMLNrgMuAC4DOwM/4va0b9/67\ngROBPkAPYGdgVMJnPAu0A3rF2vYAHkzBd0lH2+GTsS8BtpiwVVX3OPYHw+v4RPZDgXOB8/AJ4zVN\nqfc85g2K/t5nJ5zXPS+f7sB9QBfgaGAb4E0z27aggX7Xk26r9zym+vyuhxCq9QP4CLgn7rUB3wJX\nRx1buj/wJC6vlPOLgZy41w2BX4C+ca/XA7+Na9MWyAc6x163i70+KK5Nb2AT0CLqe1DF9zsfOCWK\newwcD2wEfhPX5kJgJVAn6ntTxff8ceDFUt6je175+/6b2P3pFndMv+tVf8+r1e96te4hscIN+N4u\nOBb8TpS6AZ8U0SbWtf21mT1tZrsBmFkrPJuOv7drgMkU3tuD8Yw4vs0XePG6gjaHAitDCNPiPvMt\n/F+uXVLzlaqHKr7HhwIzQwjL49qMAxoB+yXpK1UnR8a6ueeY2QNm1jTuXBa655XVGL8XK0C/61Wk\nyB1pECgAAANGSURBVD2PU21+16t1QkLpG/C1qPpwqp2P8G613sBFQCvg/2Jjgy3wX7jS7m1zYEPs\nD5aS2rQAfog/GULYjP9Pk+n/jaryHrco4XMg8/47vAEMAHoCVwNHAK+bmcXOt0D3vMJi9/Fu4P0Q\nQsGcNP2up1AJ9xyq2e96WhdGk9QKIcTvWfCZmU0BFgJ9gTnRRCWSWiGEF+JezjKzmcDXwJHAu5EE\nVbM8ALQHukYdSAYp9p5Xt9/16t5DUt4N+KQUIYTVwFygNX7/jNLv7RKgrpk13EqbxBnbtYGm6L9R\nVd7jJSV8DmT4f4cQwnz8z5L/b+/uXaMIwjiOf0dQg4oEglbRIAjaxBeMjSAqAQtB7Cz9A6ysUgtW\naiuxEGwULWxSiAiWEg8hKaxEBBULX0ARURJBwlo8e7JZNeCR27nV7we2yM2Q2/1l2H12d4Z0V3yY\neY9SSleAE8DRoijeVpoc632yQua/GPSx3uqCpCiK78A8MfMX+PnoahJ4lGu/2iqltIkYqG/KgfuO\n5dluJt4ZdrOdJyY2VfvsArYDnfKjDjCcUtpf+apJ4uT0uD9H0g4NZ9wBxmurz44Dn4FlS73/Nyml\nUWAE6J7MzbwH5YXxFHCsKIrX1TbHen+slPkf+g/2WM89M3gVZhafBhaI92S7iaVIH4Etufdt0Dfg\nMrF8aww4BDwg3vuNlO1TZZYngXFgBngOrKv8jmngJfEI8AAwCzysfc89YA44SDxSfAbcyH38DWW8\nEdgL7CNmqp8rf97WZMbEzccT4p3yHmLe0HvgQu6Mmsy8bLtEXAjHiBPrHPAUWGvmPWc+TayoOEzc\nGXe3oUofx3qDmbdxrGcPdZX+MGeBV8QSsg4wkXuf2rABt4kl0ovErOpbwI5an/PEcr0FYtb0zlr7\nemIt/AfgC3AH2FrrMwzcJKrlT8A1YEPu428o4yPERXGptl1vOmPignwX+FqeLC4Ca3Jn1GTmwBBw\nn7hb/wa8AK5Su4Ex87/O/Hd5LwFnav0c6w1l3sax7j/XkyRJ2bV6DokkSfo3WJBIkqTsLEgkSVJ2\nFiSSJCk7CxJJkpSdBYkkScrOgkSSJGVnQSJJkrKzIJEkSdlZkEiSpOwsSCRJUnY/ABKjF+31XFT2\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa8fc1ae2d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24110/150000 [1:56:09<10:01:29,  3.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24110\tepsilon=0.145\tloss=8.775\treward/tick=1.601\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24120/150000 [1:56:11<7:17:38,  4.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24120\tepsilon=0.145\tloss=10.569\treward/tick=1.596\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24130/150000 [1:56:13<7:14:37,  4.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24130\tepsilon=0.145\tloss=17.946\treward/tick=1.841\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24140/150000 [1:56:15<7:19:56,  4.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24140\tepsilon=0.145\tloss=11.187\treward/tick=1.759\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24150/150000 [1:56:17<8:16:35,  4.22it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24150\tepsilon=0.145\tloss=6.942\treward/tick=1.721\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24160/150000 [1:56:19<7:14:59,  4.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24160\tepsilon=0.144\tloss=5.296\treward/tick=1.905\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24170/150000 [1:56:21<7:11:43,  4.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24170\tepsilon=0.144\tloss=3.790\treward/tick=1.860\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24180/150000 [1:56:23<7:23:14,  4.73it/s]"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from IPython.display import clear_output\n",
    "\n",
    "\n",
    "for i in trange(150000):    \n",
    "    \n",
    "    ##update agent's epsilon (in e-greedy policy)\n",
    "    current_epsilon = 0.01 + 0.45*np.exp(-epoch_counter/20000.)\n",
    "    action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "\n",
    "    #play\n",
    "    pool.update(SEQ_LENGTH)\n",
    "\n",
    "    #train\n",
    "    loss = 0.95*loss + 0.05*train_step()\n",
    "    targetnet.load_weights(0.01)\n",
    "    \n",
    "    \n",
    "    if epoch_counter%10==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        reward_per_tick = 0.95*reward_per_tick + 0.05*pool.experience_replay.rewards.get_value().mean()\n",
    "        print(\"iter=%i\\tepsilon=%.3f\\tloss=%.3f\\treward/tick=%.3f\"%(epoch_counter,\n",
    "                                                               current_epsilon,\n",
    "                                                               loss,\n",
    "                                                               reward_per_tick))\n",
    "        \n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%100 ==0:\n",
    "        action_layer.epsilon.set_value(0)\n",
    "        reward = 0.95*reward + 0.05*np.mean(pool.evaluate(record_video=False))\n",
    "        action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "        \n",
    "        rewards[epoch_counter] = reward\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.plot(*zip(*sorted(rewards.items(),key=lambda (t,r):t)))\n",
    "        plt.show()\n",
    "        \n",
    "\n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating results\n",
    " * Here we plot learning curves and sample testimonials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "plt.plot(*zip(*sorted(rewards.items(),key=lambda k:k[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"pacman.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "###LOAD FROM HERE\n",
    "from agentnet.utils.persistence import load\n",
    "load(action_layer,\"pacman.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "action_layer.epsilon.set_value(0.01)\n",
    "rw = pool.evaluate(n_games=20,save_path=\"./records\",record_video=False)\n",
    "print(\"mean session score=%f.5\"%np.mean(rw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Once you got it working,\n",
    "Try building a network that maximizes the final score\n",
    "\n",
    "* Moar lasagne stuff: convolutional layers, batch normalization, nonlinearities and so on\n",
    "* Recurrent agent memory layers, GRUMemoryLayer, etc\n",
    "* Different reinforcement learning algorithm (p.e. qlearning_n_step), other parameters\n",
    "* Experience replay pool\n",
    "\n",
    "\n",
    "Look for info?\n",
    "* [lasagne doc](http://lasagne.readthedocs.io/en/latest/)\n",
    "* [agentnet doc](http://agentnet.readthedocs.io/en/latest/)\n",
    "* [gym homepage](http://gym.openai.com/)\n",
    "\n",
    "\n",
    "You can also try to expand to a different game: \n",
    " * all OpenAI Atari games are already compatible, you only need to change GAME_TITLE\n",
    " * Other discrete action space environments are also accessible this way\n",
    " * For continuous action spaces, either discretize actions or use continuous RL algorithms (e.g. .learning.dpg_n_step)\n",
    " * Adapting to a custom non-OpenAI environment can be done with a simple wrapper\n",
    " \n",
    " \n",
    "__Good luck!__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}