notebooks/Automated_Curriculum_Learning_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **How to run this notebook (command-line)?**\n",
    "1. Install the `ReinventCommunity` environment:\n",
    "`conda env create -f environment.yml`\n",
    "2. Activate the environment:\n",
    "`conda activate ReinventCommunity`\n",
    "3. Execute `jupyter`:\n",
    "`jupyter notebook`\n",
    "4. Copy the link to a browser\n",
    "\n",
    "\n",
    "# `REINVENT 3.2`: Automated Curriculum Learning Demo\n",
    "\n",
    "The aim of this notebook is to illustrate how `REINVENT` can be used for the _de novo_ design of molecules in a *Curriculum Learning* (CL) setup. The general idea of CL is to decompose a Production Objective (target properties for the proposed molecules) into simpler sequential Curriculum Objectives to accelerate convergence: \n",
    "\n",
    "![](img/REINVENT_CL_mode.png)\n",
    "The purpose of the Curriculum Objectives is to guide the REINVENT agent to areas of chemical space that satisfy the Production Objective. The order of Curriculum Objectives is user-defined and each Objective (Curriculum and Production) can consist of any REINVENT scoring function component(s), e.g., Tanimoto similarity. Progression through Curriculum Objectives is controlled by a threshold (can be specified for different Objectives) which the agent must achieve.\n",
    "\n",
    "In the following section, we will show how to set up an `Automated Curriculum Learning` `REINVENT` run that gradually generates compounds possessing a target scaffold (which is not present in the training set of the provided prior) that represents a difficult task for a standard `REINVENT` run. The target scaffold is dihydro-pyrazoloquinazoline and is a known active scaffold against 3-phosphoinositide-dependent protein kinase-1 (PDK1). The reference paper is:\n",
    "\n",
    "**Angiolini, M.; Banfi, P.; Casale, E.; Casuscelli, F.; Fiorelli, C.; Saccardo, M. B.; Silvagni, M.; Zuccotto, F. Structure-Based Optimization of Potent PDK1 Inhibitors. Bioorg. Med. Chem. Lett. 2010, 20 (14), 4095–4099. https://doi.org/10.1016/j.bmcl.2010.05.070.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setting the Paths\n",
    "_Please update the following code block such that it reflects your system's installation and execute it._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load dependencies\n",
    "import os\n",
    "import re\n",
    "import json\n",
    "import tempfile\n",
    "\n",
    "# --------- change these path variables as required\n",
    "reinvent_dir = os.path.expanduser(\"~/Desktop/Reinvent\")\n",
    "reinvent_env = os.path.expanduser(\"~/miniconda3/envs/reinvent.v3.2\")\n",
    "output_dir = os.path.expanduser(\"~/Desktop/REINVENT_AutoCL_demo\")\n",
    "\n",
    "# --------- do not change\n",
    "# get the notebook's root path\n",
    "try: ipynb_path\n",
    "except NameError: ipynb_path = os.getcwd()\n",
    "\n",
    "# if required, generate a folder to store the results\n",
    "try:\n",
    "    os.mkdir(output_dir)\n",
    "except FileExistsError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setting the Configuration\n",
    "In the cells below we will build a nested dictionary object that will be eventually converted to JSON file which in turn will be interpreted by `REINVENT`. \n",
    "You can find this file in your `output_dir` location."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A) Declare the Run Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the dictionary\n",
    "configuration = {\n",
    "    \"version\": 3,                                 # we are going to use REINVENT's newest release\n",
    "    \"run_type\": \"curriculum_learning\",            # other run types: \"sampling\", \"reinforcement_learning\",\n",
    "                                                  #                  \"transfer_learning\",\n",
    "                                                  #                  \"scoring\" and \"create_model\"\n",
    "    \"model_type\": \"default\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B) Sort out the logging details\n",
    "This includes `result_folder` path where the results will be produced.\n",
    "\n",
    "Also: `REINVENT` can send custom log messages to a remote location. We have retained this capability in the code. if the `recipient` value differs from `\"local\"`, `REINVENT` will attempt to POST the data to the specified `recipient`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add block to specify whether to run locally or not and\n",
    "# where to store the results and logging\n",
    "configuration[\"logging\"] = {\n",
    "    \"sender\": \"http://0.0.0.1\",            # only relevant if \"recipient\" is set to \"remote\"\n",
    "    \"recipient\": \"local\",                  # either to local logging or use a remote REST-interface\n",
    "    \"logging_frequency\": 100,              # log every x-th steps\n",
    "    \"logging_path\": os.path.join(output_dir, \"progress.log\"), # load this folder in tensorboard\n",
    "    \"result_folder\": os.path.join(output_dir, \"results\"),     # will hold the compounds (SMILES) and summaries\n",
    "    \"job_name\": \"Automated Curriculum Learning Demo\",         # set an arbitrary job name for identification\n",
    "    \"job_id\": \"Demo\"                       # only relevant if \"recipient\" is set to a specific REST endpoint\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create `parameters` field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the \"parameters\" block\n",
    "configuration[\"parameters\"] = {}\n",
    "\n",
    "# First add the paths to the Prior, Agent, and set the curriculum type to automated\n",
    "configuration[\"parameters\"][\"prior\"] = os.path.join(ipynb_path, \"models/random.prior.new\")\n",
    "configuration[\"parameters\"][\"agent\"] = os.path.join(ipynb_path, \"models/random.prior.new\")\n",
    "configuration[\"parameters\"][\"curriculum_type\"] = \"automated\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C) Specify the Curriculum Strategy\n",
    "Overview of important `REINVENT` parameters:\n",
    "* **Diversity Filter**: If the agent becomes very focussed, it tends to produce the similar molecules over and over (because they return high scores). To enrich different scaffolds, we can activate the diversity filter, which will \"bin\" the molecules into groups (scaffolds). Once a given bin is full, all other molecules with the same scaffold will be penalized score-wise, effectively \"pushing\" the agent out of a local minimum in the score landscape thus enriching diversity. \n",
    "* **Inception**: Sometimes agents \"linger around\" for a while before they (by chance) happen to pick up a trace and generate interesting compounds. To speed up this very early exploration, we can *incept* a couple of promising molecules as list of `SMILES`. Inception also allows storing of molecules up to `memory_size` which correspond to the highest scoring molecules which can be replayed back to the agent to keep it on track.\n",
    "\n",
    "These 2 parameters are relevant for any `REINVENT` experiment but there are additional features and considerations for a Curriculum Learning experiment. Curriculum Learning is split into a Curriculum Phase and Production Phase. During the Curriculum Phase, Curriculum Objectives are used to guide the agent to favourable chemical space. During the Production Phase, the Production Objective is activated and the agent samples for a pre-defined number of epochs, presumably generating favourable molecules. \n",
    "1) A separate and distinct **Diversity Filter** can be specified in the Curriculum and Production Phases. This is particularly relevant as perhaps a **Diversity Filter** is not desired during the Curriculum Phase since the goal is to guide the agent to favourable chemical space and not necessarily generate molecules that satisfy the Production Objective (target objective). Conversely, once the Production Phase starts, initializing a **Diversity Filter** can ensure agent sampling of diverse minima to balance exploration and exploitation (Setting up a **Diversity Filter** for the Production Phase will be shown in the \"Specify the Production Strategy\" section)\n",
    "\n",
    "2) A separate and distinct **Inception** can also be specified in the Curriculum and Production Phases. A relevant use case is that any stored molecules in the **Inception** memory during the Curriculum Phase may not be relevant in the Production Phase. In this case, initializing a new **Inception** will clear the memory. \n",
    "\n",
    "The below cell block will set up the Curriculum Strategy which provides all the parameters necessary in the Curriculum Phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up the Curriculum Strategy\n",
    "configuration[\"parameters\"][\"curriculum_strategy\"] = {\n",
    "    \"name\": \"user_defined\",         # denotes that the order of Curriculum Objectives is defined by the user\n",
    "    \"max_num_iterations\": 1500,     # denotes the total number of epochs to spend in the Curriculum Phase\n",
    "                                    # if by the end of the total epochs the last Curriculum Objective is not \n",
    "                                    # satisfied (based on the agent achieving a score >= threshold), the run stops\n",
    "    \"batch_size\": 128,              # specifies how many molecules are generated per epoch\n",
    "    \"learning_rate\": 0.0001,        # sets how strongly the agent is influenced by each epoch\n",
    "    \"sigma\": 128,                   # used to calculate the \"augmented likelihood\", see publication\n",
    "    \"learning_strategy\": {\n",
    "        \"name\": \"dap_single_query\",\n",
    "        \"parameters\": {\n",
    "            \"sigma\": 120\n",
    "        }\n",
    "    },\n",
    "    \"diversity_filter\": {\n",
    "        \"name\": \"NoFilter\",         # other options are: \"IdenticalTopologicalScaffold\", \n",
    "                                    #                    \"IdenticalMurckoScaffold\", and \"ScaffoldSimilarity\"\n",
    "\n",
    "        \"bucket_size\": 25,          # the bin size; penalization will start once this is exceeded\n",
    "        \"minscore\": 0.4,            # the minimum total score to be considered for binning\n",
    "        \"minsimilarity\": 0.4        # the minimum similarity to be placed into the same bin\n",
    "    },\n",
    "    \"inception\": {\n",
    "        \"smiles\": [],               # fill in a list of SMILES here that can be used (or leave empty)\n",
    "        \"memory_size\": 100,         # sets how many molecules are to be remembered\n",
    "        \"sample_size\": 10           # how many are to be sampled each epoch from the memory for experience replay\n",
    "    },\n",
    "    # Curriculum Objectives are all the scoring functions that are to be sequentially activated\n",
    "    \"curriculum_objectives\": [{\n",
    "        # 1st scoring function below\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",     # this is our default one (alternative: \"custom_sum\")\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",     # enforce the match to a given substructure\n",
    "                \"name\": \"Pyrimidine\",     # arbitrary name for the component\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[c]1[c][c]n[c]n1\"     # a match with this substructure is required\n",
    "                    ]\n",
    "                },    \n",
    "                \"weight\": 1}]             # the weight of the component (default: 1)\n",
    "            },\n",
    "        \"score_threshold\": 0.8            # agent must achieve an average score of this before \n",
    "                                          # progressing to the next Curriculum Objective \n",
    "        },\n",
    "        # 2nd scoring function below\n",
    "        {\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"H-Bonding Ring\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[c]1[c][c]nc(n1)[N]\"\n",
    "                    ]\n",
    "                },\n",
    "                \"weight\": 1}]\n",
    "            },\n",
    "        \"score_threshold\": 0.8\n",
    "        },\n",
    "        # 3rd scoring function below\n",
    "        {\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"H-Bonding Ring with Phenyl\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[c]1[c][c]c([c][c]1)[N]c2n[c][c][c]n2\"\n",
    "                    ]\n",
    "                },\n",
    "                \"weight\": 1}]\n",
    "            },\n",
    "        \"score_threshold\": 0.8\n",
    "        },\n",
    "        # 4th scoring function below\n",
    "        {\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"Double Ring\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[c]1[c][c]c([c][c]1)[N]c2n[c]c3c(n2)-[c][c][C][C]3\"\n",
    "                    ]\n",
    "                },    \n",
    "                \"weight\": 1}]\n",
    "            },\n",
    "        \"score_threshold\": 0.8\n",
    "        },\n",
    "        # 5th scoring function below\n",
    "        {\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"Triple Ring\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[c]1[c][c]c([c][c]1)[N]c2n[c]c3c(n2)-c4c([c]n[n]4)[C][C]3\"\n",
    "                    ]\n",
    "                },\n",
    "                \"weight\": 1}]\n",
    "            },\n",
    "        \"score_threshold\": 0.8\n",
    "        },\n",
    "        # 6th scoring function below\n",
    "        {\n",
    "        \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"Full Substructure\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[*]NC(=O)c1nn([*])c2c1CCc3cnc(Nc4ccccc4)nc23\"\n",
    "                    ]\n",
    "                },\n",
    "                \"weight\": 1}]\n",
    "            },\n",
    "        \"score_threshold\": 0.8\n",
    "        },\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D) Specify the Production Strategy\n",
    "\n",
    "The Production Strategy provides the parameters to be used in the Production Phase. Here, the Production Objective (target objective) is activated. A new **Diversity Filter** and **Inception** can be initialized. We keep the Curriculum Phase **Inception** here as the last Curriculum Objective is the same as the Production Objective. Moreover, we continue to use **NoFilter** so the agent is not penalized for sampling the same scaffold as our only goal in this tutorial is to generate the target scaffold. Using a **Diversity Filter** would penalize the agent and eventually give compounds possesing the target scaffold a score of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up the Curriculum Strategy\n",
    "configuration[\"parameters\"][\"production_strategy\"] = {\n",
    "    \"name\": \"standard\",\n",
    "    \"retain_inception\": True,       # option to retain the inception from the Curriculum Phase\n",
    "                                    # retain it here since the last Curriculum Objective is the same as\n",
    "                                    # Production Objective. Previous top compounds will be relevant\n",
    "    \n",
    "    \"number_of_steps\": 100,         # number of epochs to run the Production Phase\n",
    "    \"batch_size\": 128,              # specifies how many molecules are generated per epoch\n",
    "    \"learning_rate\": 0.0001,        # sets how strongly the agent is influenced by each epoch\n",
    "    \"sigma\": 128,                   # used to calculate the \"augmented likelihood\", see publication\n",
    "    \"learning_strategy\": {\n",
    "        \"name\": \"dap_single_query\",\n",
    "        \"parameters\": {\n",
    "            \"sigma\": 120\n",
    "        }\n",
    "    },\n",
    "    \"diversity_filter\": {\n",
    "        \"name\": \"NoFilter\",         # other options are: \"IdenticalTopologicalScaffold\",\n",
    "                                    #                    \"IdenticalMurckoScaffold\"\", and \"ScaffoldSimilarity\"\n",
    "\n",
    "        \"bucket_size\": 25,          # the bin size; penalization will start once this is exceeded\n",
    "        \"minscore\": 0.4,            # the minimum total score to be considered for binning\n",
    "        \"minsimilarity\": 0.4        # the minimum similarity to be placed into the same bin\n",
    "    },\n",
    "    \"inception\": {\n",
    "        \"smiles\": [],               # fill in a list of SMILES here that can be used (or leave empty)\n",
    "        \"memory_size\": 100,         # sets how many molecules are to be remembered\n",
    "        \"sample_size\": 10           # how many are to be sampled each epoch from the memory for experience replay\n",
    "    },\n",
    "    # the Production Objective contains the final scoring function to be activated\n",
    "    # here, it is the same scoring function as the last Curriculum Objective\n",
    "    # as we want to continue sampling the target substructure\n",
    "    \"scoring_function\": {\n",
    "            \"name\": \"custom_product\",\n",
    "            \"parallel\": False,\n",
    "            \"parameters\": [{\n",
    "                \"component_type\": \"matching_substructure\",\n",
    "                \"name\": \"Full Substructure\",\n",
    "                \"specific_parameters\": {\n",
    "                    \"smiles\": [\n",
    "                            \"[*]NC(=O)c1nn([*])c2c1CCc3cnc(Nc4ccccc4)nc23\"\n",
    "                    ]\n",
    "                },\n",
    "                \"weight\": 1}]\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write Out the Configuration\n",
    "We now have successfully filled the dictionary and will write it out as a `JSON` file in the output directory. Please have a look at the file before proceeding in order to see how the paths have been inserted where required and the `dict` -> `JSON` translations (e.g. `True` to `true`) have taken place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "configuration_JSON_path = os.path.join(output_dir, \"AutoCL_config.json\")\n",
    "with open(configuration_JSON_path, 'w') as f:\n",
    "    json.dump(configuration, f, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Run `REINVENT`\n",
    "Now it is time to execute `REINVENT` locally. Note, that depending on the number of epochs (steps) and the execution time of the scoring function components, this might take a while. The \"matching_substructure\" component should be fairly quick, and the total runtime should be under 15 minutes.\n",
    "\n",
    "**Note**: Sometimes, `REINVENT` will be unsuccessful in generating the desired substructure in this demo as there is stochasticity involved in the sampling process. The substructure was not present in the training set for the prior and thus represents a challenging task.\n",
    "\n",
    "The command-line execution looks like this:\n",
    "```\n",
    "# activate envionment\n",
    "conda activate reinvent.v3.0\n",
    "\n",
    "# execute REINVENT\n",
    "python <your_path>/input.py <config>.json\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%capture captured_err_stream --no-stderr\n",
    "\n",
    "# execute REINVENT from the command-line\n",
    "!{reinvent_env}/bin/python {reinvent_dir}/input.py {configuration_JSON_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyse The Results\n",
    "In order to analyze the run, we can use `tensorboard`:\n",
    "\n",
    "```\n",
    "# go to the root folder of the output\n",
    "cd <your_path>/REINVENT_AutoCL_demo\n",
    "\n",
    "# make sure you have activated the proper environment\n",
    "conda activate reinvent.v3.2\n",
    "\n",
    "# start tensorboard\n",
    "tensorboard --logdir progress.log\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "Then copy the link provided to a browser window, e.g. \"http://workstation.url.com:6006/\". The following figures are example plots - remember, that there is always some randomness involved so replicate runs will follow different training progressions.\n",
    "\n",
    "**Note: There is a chance that the curriculum learning run does not find the target scaffold. The target scaffold is not present in the prior and thus may not be found in 1500 epochs (as enforced in the configuration `JSON`)**\n",
    "\n",
    "In `tensorboard` you can monitor the individual scoring function components. By analyzing the average score plot, we can see that the agent gradually constructs the target scaffold. The plot is annotated with the substructures. We further observe that the `Fraction_valid_SMILES` was high throughout. \n",
    "\n",
    "![](img/AutoCL_Training_Plots.png)\n",
    "\n",
    "There is also an \"Images\" tab available in `tensorboard` that lets you browse through the compounds generated in an easy way. In the moleculess, the target scaffold is highlighted in red (if present). Also, the total scores are given per molecule. Below is what was observed for epoch 1060. The generated compounds feature the target scaffold and therefore all possess the maximum score of 1.000.\n",
    "\n",
    "![](img/AutoCL_Sample_Compounds.png)\n",
    "\n",
    "Finally, scaffold memories is a `CSV` file containing all the compounds collected with each each Curriculum Objective or Production Objective activated. All Curriculum Objective scaffold memories are identified by a number suffix starting from 0 (denoting the first Curriculum Objective). The scaffold memory for the Production Objective is `scaffold_memory.csv`. The files are saved at:\n",
    "\n",
    "`<your_path>/REINVENT_AutoCL_demo/results`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}