complex qa/recursive_retriever_document_agent_subquestion_query_qa_llamaindex.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complex Query Resolution through LlamaIndex Utilizing Recursive Retrieval, Document Agents, and Sub Question Query Decomposition"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we experiment with the aim of resolving complex queries by leveraging LlamaIndex, integrating Recursive Retrieval, Document Agents, and Sub Question Query Engine. We navigate through a structured pathway using 3 query engines as tools for Document Agents - Vector Index, Summary Index, and Knowledge Graph Index ensuring that the system adeptly manages multifaceted inquiries, providing coherent and contextually rich responses by seamlessly integrating and synthesizing information from diverse documents and data sources."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![llamaindex-rr-da-sqe.png](../assets/img/llamaindex-rr-da-sqe.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by installing the dependencies and importing the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dHOIM9LgEe_P"
   },
   "outputs": [],
   "source": [
    "%pip install llama-index pinecone-client transformers neo4j python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "TNjKt1FvVOvF"
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "ttqtgRD4CV1B"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import torch\n",
    "import pinecone\n",
    "from transformers import pipeline\n",
    "from llama_index import (\n",
    "    VectorStoreIndex,\n",
    "    SummaryIndex,\n",
    "    KnowledgeGraphIndex,\n",
    "    SimpleKeywordTableIndex,\n",
    "    SimpleDirectoryReader,\n",
    "    ServiceContext,\n",
    "    StorageContext\n",
    ")\n",
    "from dotenv import load_dotenv\n",
    "from llama_index.schema import IndexNode\n",
    "from llama_index.tools import QueryEngineTool, ToolMetadata\n",
    "from llama_index.llms import OpenAI\n",
    "from llama_index.query_engine import SubQuestionQueryEngine\n",
    "from llama_index.retrievers import RecursiveRetriever\n",
    "from llama_index.query_engine import RetrieverQueryEngine\n",
    "from llama_index.response_synthesizers import get_response_synthesizer\n",
    "from llama_index.vector_stores import PineconeVectorStore\n",
    "from llama_index.graph_stores import Neo4jGraphStore"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "71CsIAA36Aur"
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work with 3 Wikipedia pages. Extract the data, store it, and finally load it for further processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "cbX6NmoZEdNw"
   },
   "outputs": [],
   "source": [
    "wiki_titles = [\"Seattle\", \"Boston\", \"Chicago\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "ria230SREdLJ"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import requests\n",
    "\n",
    "for title in wiki_titles:\n",
    "    response = requests.get(\n",
    "        \"https://en.wikipedia.org/w/api.php\",\n",
    "        params={\n",
    "            \"action\": \"query\",\n",
    "            \"format\": \"json\",\n",
    "            \"titles\": title,\n",
    "            \"prop\": \"extracts\",\n",
    "            # 'exintro': True,\n",
    "            \"explaintext\": True,\n",
    "        },\n",
    "    ).json()\n",
    "    page = next(iter(response[\"query\"][\"pages\"].values()))\n",
    "    wiki_text = page[\"extract\"]\n",
    "\n",
    "    data_path = Path(\"data\")\n",
    "    if not data_path.exists():\n",
    "        Path.mkdir(data_path)\n",
    "\n",
    "    with open(data_path / f\"{title}.txt\", \"w\") as fp:\n",
    "        fp.write(wiki_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "vvNYPZRREdIq"
   },
   "outputs": [],
   "source": [
    "# Load all wiki documents\n",
    "city_docs = {}\n",
    "for wiki_title in wiki_titles:\n",
    "    city_docs[wiki_title] = SimpleDirectoryReader(\n",
    "        input_files=[f\"data/{wiki_title}.txt\"]\n",
    "    ).load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "32Sm8vfCSQQZ"
   },
   "outputs": [],
   "source": [
    "load_dotenv()\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GMNj2l4dEdGF",
    "outputId": "06969219-c7af-49fe-a6e6-374d8916ca94"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /tmp/llama_index...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\n",
    "service_context = ServiceContext.from_defaults(llm=llm)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "wUTPv_iM6KTT"
   },
   "source": [
    "## Build Document Agent for each Document"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define document agents for each document.\n",
    "Before that we define - a vector index (for semantic search), summary index (for summarization), and a graph index (for structural semantic search) for each document. These 3 query engines are then converted into tools that are passed to an OpenAI function calling agent.\n",
    "This document agent can dynamically choose to perform semantic search over vector index or graph index or summarization within a given document.\n",
    "We create a separate document agent for each city."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "jXD6hTArRBlx"
   },
   "source": [
    "### Vector Storage Context"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a Pinecone vector index with specified parameters like dimension and metric. And a vector storage context is established, utilizing Pinecone's vector store, to manage and facilitate the efficient storage and retrieval of the vector index data within the LlamaIndex framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "l6McX81LNvyI"
   },
   "outputs": [],
   "source": [
    "# init pinecone\n",
    "os.environ[\"PINECONE_API_KEY\"] = os.getenv('PINECONE_API_KEY')\n",
    "os.environ[\"PINECONE_ENVIRONMENT\"] = os.getenv('PINECONE_ENVIRONMENT')\n",
    "pinecone.init(api_key=os.environ[\"PINECONE_API_KEY\"], environment=os.environ[\"PINECONE_ENVIRONMENT\"])\n",
    "pinecone.create_index(\"vector-index\", dimension=1536, metric=\"euclidean\", pod_type=\"p1\")\n",
    "\n",
    "# construct vector store and customize storage context\n",
    "vector_storage_context = StorageContext.from_defaults(\n",
    "    vector_store=PineconeVectorStore(pinecone.Index(\"vector-index\"))\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ooMJEoOURBCx"
   },
   "source": [
    "### Graph Storage Context"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we build a knowledge graph from scratch using Relation Extraction By End-to-end Language generation (REBEL), LlamaIndex, and Neo4j. REBEL is a relation extraction model which uses a BART model to convert raw sentences into relation triplets. We essentially construct a knowledge graph from unstructured data for efficient granular knowledge retrieval. Lastly, we utilize Neo4j's graph store, to manage and facilitate the efficient storage and retrieval of the graph data within the LlamaIndex framework."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![p4-kg.png](../assets/img/p4-kg.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "K5pBj5ZgRB2h"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06c59b04b9e146d7bbfa5c82df2ed0bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/1.42k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2bb1198d085488fa89f5c2576710f2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading model.safetensors:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57b6329c104a418e9dcc19f9a3eaa557",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/1.23k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "051a56332b7c482b9459f0ea2c46a3bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/vocab.json:   0%|          | 0.00/798k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d9ab9dcad34446ba820f8b96cf6ef1f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c1175f2814c485a99593fe06d0d7fc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05f4765bd0844c648ecbdb64e526dab8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)in/added_tokens.json:   0%|          | 0.00/123 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d515e8c3efc14118998f9fdc22f88898",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/344 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)\n",
    "\n",
    "triplet_extractor = pipeline(\n",
    "    'text2text-generation',\n",
    "    model='Babelscape/rebel-large',\n",
    "    tokenizer='Babelscape/rebel-large',\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "IyRS4UlqRLdc"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def clean_triplets(input_text, triplets):\n",
    "    \"\"\"Sometimes the model hallucinates, so we filter out entities\n",
    "       not present in the text\"\"\"\n",
    "    text = input_text.lower()\n",
    "    clean_triplets = []\n",
    "    for triplet in triplets:\n",
    "\n",
    "        if (triplet[\"head\"] == triplet[\"tail\"]):\n",
    "            continue\n",
    "\n",
    "        head_match = re.search(\n",
    "            r'\\b' + re.escape(triplet[\"head\"].lower()) + r'\\b', text)\n",
    "        if head_match:\n",
    "            head_index = head_match.start()\n",
    "        else:\n",
    "            head_index = text.find(triplet[\"head\"].lower())\n",
    "\n",
    "        tail_match = re.search(\n",
    "            r'\\b' + re.escape(triplet[\"tail\"].lower()) + r'\\b', text)\n",
    "        if tail_match:\n",
    "            tail_index = tail_match.start()\n",
    "        else:\n",
    "            tail_index = text.find(triplet[\"tail\"].lower())\n",
    "\n",
    "        if ((head_index == -1) or (tail_index == -1)):\n",
    "            continue\n",
    "\n",
    "        clean_triplets.append((triplet[\"head\"], triplet[\"type\"], triplet[\"tail\"]))\n",
    "\n",
    "    return clean_triplets\n",
    "\n",
    "def extract_triplets(input_text):\n",
    "    text = triplet_extractor.tokenizer.batch_decode([triplet_extractor(input_text, return_tensors=True, return_text=False)[0][\"generated_token_ids\"]])[0]\n",
    "\n",
    "    triplets = []\n",
    "    relation, subject, relation, object_ = '', '', '', ''\n",
    "    text = text.strip()\n",
    "    current = 'x'\n",
    "    for token in text.replace(\"<s>\", \"\").replace(\"<pad>\", \"\").replace(\"</s>\", \"\").split():\n",
    "        if token == \"<triplet>\":\n",
    "            current = 't'\n",
    "            if relation != '':\n",
    "                triplets.append({'head': subject.strip(), 'type': relation.strip(),'tail': object_.strip()})\n",
    "                relation = ''\n",
    "            subject = ''\n",
    "        elif token == \"<subj>\":\n",
    "            current = 's'\n",
    "            if relation != '':\n",
    "                triplets.append({'head': subject.strip(), 'type': relation.strip(),'tail': object_.strip()})\n",
    "            object_ = ''\n",
    "        elif token == \"<obj>\":\n",
    "            current = 'o'\n",
    "            relation = ''\n",
    "        else:\n",
    "            if current == 't':\n",
    "                subject += ' ' + token\n",
    "            elif current == 's':\n",
    "                object_ += ' ' + token\n",
    "            elif current == 'o':\n",
    "                relation += ' ' + token\n",
    "\n",
    "    if subject != '' and relation != '' and object_ != '':\n",
    "        triplets.append({'head': subject.strip(), 'type': relation.strip(), 'tail':object_.strip()})\n",
    "    clean = clean_triplets(input_text, triplets)\n",
    "    return clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "KMgp-JhHRMa8"
   },
   "outputs": [],
   "source": [
    "os.environ[\"NEO4J_URI\"] = os.getenv('NEO4J_URI')\n",
    "os.environ[\"NEO4J_USERNAME\"] = os.getenv('NEO4J_USERNAME')\n",
    "os.environ[\"NEO4J_PASSWORD\"] = os.getenv('NEO4J_PASSWORD')\n",
    "os.environ[\"NEO4J_DB\"] = os.getenv('NEO4J_DB')\n",
    "\n",
    "graph_store = Neo4jGraphStore(\n",
    "    username=os.environ[\"NEO4J_USERNAME\"],\n",
    "    password=os.environ[\"NEO4J_PASSWORD\"],\n",
    "    url=os.environ[\"NEO4J_URI\"],\n",
    "    database=os.environ[\"NEO4J_DB\"],\n",
    ")\n",
    "\n",
    "graph_storage_context = StorageContext.from_defaults(graph_store=graph_store)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 168,
     "referenced_widgets": [
      "a956afde996449319b10ddb082bb8fc2",
      "ddf3f76146984a8596ec8e734f9b2767",
      "1531d48a1a694db29793bd7db8596346",
      "11413f3af9574736a5a52cd8f5582778",
      "73d9bfd4f94f44ed9a472a101be84a0b",
      "9b198221deb4448cb433370898fbe243",
      "9a13a789d9e04d20b3e98343832ff7e0",
      "856d8052231246d9bf0c859379a8c11f",
      "2e54f4d63e0048dfbb2414c712d3fa0d",
      "ad697e475cb14f1c8c201209511d4e3a",
      "8d5d68c4bde74c3bbf0ed61182728d0f",
      "89ece7effef745218c0ae340cb49459d",
      "9dbbeac38c304324878830799c49f090",
      "b288f9667a7d4240bab98b0390326cd9",
      "7c32e2ba7ace4d158695d6375aead04d",
      "128cc0fdcab64227a9f149e290f1a94f",
      "9300a17d34154ffb87d819cc869afb6d",
      "798981bae72742e6bc59c5ed89a20d68",
      "d765e261007c4a618f8ed209ef8d97ed",
      "9cc2a6a3ed3649dbadd7e0ab8d45b573",
      "2049d632a953467fb48f14ec63a88261",
      "7b41f3c50aa646af812e3c1660724942",
      "0e93a2479384409a87ec321dafc37a27",
      "31b34df9ba6a445d93cc6233e3f4b824",
      "78e44903bbf34a338ebaf98b0ad7a090",
      "c528a5c26e0a493d9cbadec7b61eb0e2",
      "78c6176c32a047869fb0912271fcab75",
      "6f74f396370b4e3e92f8634d20f59546",
      "1d28c3107cf04d42bd2c08c6f5b5c55f",
      "870b5a6f264549ebae86c611abc4e5ed",
      "181218d0f5514e1a8346da804656f5d0",
      "27439087326c40d8bf37a400fdfe6705",
      "f944441a44c944459a45e694c43d6c8b"
     ]
    },
    "id": "VJeZahd1EdD6",
    "outputId": "87adf785-3596-4552-eaac-edebbc9ee5ed"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "08ec3bd269cd45018527fb30b1fa3e4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upserted vectors:   0%|          | 0/17 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a570f3cf31e54d3c87a4c117fefba464",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upserted vectors:   0%|          | 0/17 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9873d60d4a6a423791e6e80b5938c73a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upserted vectors:   0%|          | 0/23 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.agent import OpenAIAgent\n",
    "\n",
    "# Build agents dictionary\n",
    "agents = {}\n",
    "\n",
    "for wiki_title in wiki_titles:\n",
    "    # build vector index\n",
    "    vector_index = VectorStoreIndex.from_documents(\n",
    "        city_docs[wiki_title], service_context=service_context, storage_context=vector_storage_context\n",
    "    )\n",
    "    # build summary index\n",
    "    summary_index = SummaryIndex.from_documents(\n",
    "        city_docs[wiki_title], service_context=service_context\n",
    "    )\n",
    "    graph_index = KnowledgeGraphIndex.from_documents(\n",
    "        city_docs[wiki_title],\n",
    "        storage_context=graph_storage_context,\n",
    "        kg_triplet_extract_fn=extract_triplets,\n",
    "        service_context=ServiceContext.from_defaults(llm=llm, chunk_size=256)\n",
    "    )\n",
    "    # define query engines\n",
    "    vector_query_engine = vector_index.as_query_engine()\n",
    "    list_query_engine = summary_index.as_query_engine()\n",
    "    graph_query_engine = graph_index.as_query_engine()\n",
    "\n",
    "    # define tools\n",
    "    query_engine_tools = [\n",
    "        QueryEngineTool(\n",
    "            query_engine=vector_query_engine,\n",
    "            metadata=ToolMetadata(\n",
    "                name=\"vector_tool\",\n",
    "                description=f\"Useful for retrieving specific context from {wiki_title}\",\n",
    "            ),\n",
    "        ),\n",
    "        QueryEngineTool(\n",
    "            query_engine=list_query_engine,\n",
    "            metadata=ToolMetadata(\n",
    "                name=\"summary_tool\",\n",
    "                description=f\"Useful for summarization questions related to {wiki_title}\",\n",
    "            ),\n",
    "        ),\n",
    "        QueryEngineTool(\n",
    "            query_engine=graph_query_engine,\n",
    "            metadata=ToolMetadata(\n",
    "                name=\"graph_tool\",\n",
    "                description=f\"Useful for retrieving structural, interconnected and relational knowledge related to {wiki_title}\",\n",
    "            ),\n",
    "        ),\n",
    "    ]\n",
    "\n",
    "    # build agent\n",
    "    function_llm = OpenAI(model=\"gpt-3.5-turbo-0613\")\n",
    "    agent = OpenAIAgent.from_tools(\n",
    "        query_engine_tools,\n",
    "        llm=function_llm,\n",
    "        verbose=True,\n",
    "    )\n",
    "\n",
    "    agents[wiki_title] = agent"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "_HKRurrz6PNV"
   },
   "source": [
    "## Build Recursive Retriever over these Agents"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A set of summary nodes is established, each correlating to a respective Wikipedia city article. Subsequently, a RecursiveRetriever is configured in front these nodes, orchestrating the routing of queries to an appropriate node. This node, in turn, directs the query to the pertinent document agent, ensuring a structured pathway for query navigation and retrieval within the system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "gA4aKxfHEdBU"
   },
   "outputs": [],
   "source": [
    "# define top-level nodes\n",
    "nodes = []\n",
    "for wiki_title in wiki_titles:\n",
    "    wiki_summary = (\n",
    "        f\"This content contains Wikipedia articles about {wiki_title}. \"\n",
    "        f\"Use this index if you need to lookup specific facts about {wiki_title}.\\n\"\n",
    "        \"Do not use this index if you want to analyze multiple cities.\"\n",
    "    )\n",
    "    node = IndexNode(text=wiki_summary, index_id=wiki_title)\n",
    "    nodes.append(node)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "sG6y8N6tEc-5"
   },
   "outputs": [],
   "source": [
    "# define top-level retriever\n",
    "top_vector_index = VectorStoreIndex(nodes)\n",
    "vector_retriever = top_vector_index.as_retriever(similarity_top_k=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "RmPwf6NjEc3m"
   },
   "outputs": [],
   "source": [
    "# define recursive retriever\n",
    "recursive_retriever = RecursiveRetriever(\n",
    "    \"vector\",\n",
    "    retriever_dict={\"vector\": vector_retriever},\n",
    "    query_engine_dict=agents,\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "response_synthesizer = get_response_synthesizer(\n",
    "    response_mode=\"compact\",\n",
    ")\n",
    "retriever_query_engine = RetrieverQueryEngine.from_args(\n",
    "    recursive_retriever,\n",
    "    response_synthesizer=response_synthesizer,\n",
    "    service_context=service_context,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "vlDR-AGF6VOk"
   },
   "source": [
    "## Setup sub question query engine"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this pivotal step, the recursive retriever, encapsulating the document agent and various query engines, is transformed into a tool. This tool, characterized by its ability to access documents through recursive retrieval, is subsequently integrated into the SubQuestionQueryEngine. This ensures that the engine not only inherits the capabilities of the recursive retriever but also leverages its functionalities to effectively decompose and navigate through complex, multi-faceted queries, providing a structured and efficient pathway for extracting and synthesizing information from multiple documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "-6xTsTiIEcyc"
   },
   "outputs": [],
   "source": [
    "# convert the recursive retriever into a tool\n",
    "query_engine_tools = [\n",
    "    QueryEngineTool(\n",
    "        query_engine=retriever_query_engine,\n",
    "        metadata=ToolMetadata(\n",
    "            name=\"recursive_retriever\",\n",
    "            description=\"Recursive retriever for accessing documents\"\n",
    "        ),\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "EMvz44n9Ecv2"
   },
   "outputs": [],
   "source": [
    "# setup sub question query engine\n",
    "query_engine = SubQuestionQueryEngine.from_defaults(\n",
    "    query_engine_tools=query_engine_tools,\n",
    "    service_context=service_context,\n",
    "    use_async=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LQnTUMf5U_ts",
    "outputId": "7da725b1-2dd8-44c7-97a0-0908fb2fc7d5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 2 sub questions.\n",
      "\u001b[1;3;38;2;237;90;200m[recursive_retriever] Q: What are the sports teams in Boston?\n",
      "\u001b[0m\u001b[1;3;34mRetrieving with query id None: What are the sports teams in Boston?\n",
      "\u001b[0m\u001b[1;3;38;5;200mRetrieved node with id, entering: Boston\n",
      "\u001b[0m\u001b[1;3;34mRetrieving with query id Boston: What are the sports teams in Boston?\n",
      "\u001b[0m=== Calling Function ===\n",
      "Calling function: vector_tool with args: {\n",
      "  \"input\": \"sports teams in Boston\"\n",
      "}\n",
      "Got output: Boston has teams in the four major North American men's professional sports leagues, which are Major League Baseball, the National Football League, the National Basketball Association, and the National Hockey League. Additionally, Boston has a team in Major League Soccer.\n",
      "========================\n",
      "\u001b[1;3;32mGot response: The sports teams in Boston include:\n",
      "\n",
      "1. Boston Red Sox: The Boston Red Sox are a professional baseball team and a member of Major League Baseball (MLB). They play their home games at Fenway Park, which is the oldest ballpark in MLB.\n",
      "\n",
      "2. New England Patriots: The New England Patriots are a professional football team and a member of the National Football League (NFL). They play their home games at Gillette Stadium in Foxborough, Massachusetts.\n",
      "\n",
      "3. Boston Celtics: The Boston Celtics are a professional basketball team and a member of the National Basketball Association (NBA). They play their home games at TD Garden.\n",
      "\n",
      "4. Boston Bruins: The Boston Bruins are a professional ice hockey team and a member of the National Hockey League (NHL). They also play their home games at TD Garden.\n",
      "\n",
      "5. New England Revolution: The New England Revolution is a professional soccer team and a member of Major League Soccer (MLS). They play their home games at Gillette Stadium.\n",
      "\n",
      "These are the major sports teams in Boston, representing baseball, football, basketball, hockey, and soccer.\n",
      "\u001b[0m\u001b[1;3;38;2;90;149;237m[recursive_retriever] Q: What are the positive aspects of Seattle?\n",
      "\u001b[0m\u001b[1;3;34mRetrieving with query id None: What are the positive aspects of Seattle?\n",
      "\u001b[0m\u001b[1;3;38;5;200mRetrieved node with id, entering: Seattle\n",
      "\u001b[0m\u001b[1;3;34mRetrieving with query id Seattle: What are the positive aspects of Seattle?\n",
      "\u001b[0m=== Calling Function ===\n",
      "Calling function: summary_tool with args: {\n",
      "\"input\": \"positive aspects of Seattle\"\n",
      "}\n",
      "Got output: Seattle offers a vibrant urban environment with a thriving economy, especially in the technology sector, and is home to major companies like Microsoft and Amazon. The city also has a rich cultural scene, with a strong musical heritage in jazz and rock music. Additionally, Seattle is surrounded by breathtaking natural landscapes, including the stunning Puget Sound, Lake Washington, and the majestic Cascade Range.\n",
      "========================\n",
      "\u001b[1;3;32mGot response: Some positive aspects of Seattle include:\n",
      "\n",
      "1. Thriving Economy: Seattle is known for its strong economy, particularly in the technology sector. It is home to major companies like Microsoft, Amazon, and Boeing, providing numerous job opportunities and contributing to the city's prosperity.\n",
      "\n",
      "2. Cultural Scene: Seattle has a vibrant cultural scene with a rich musical heritage. The city has been a hub for jazz and rock music, producing famous artists like Jimi Hendrix and Nirvana. It also has a thriving theater, art, and film community, offering a diverse range of cultural experiences.\n",
      "\n",
      "3. Natural Beauty: Seattle is surrounded by stunning natural landscapes. The city is located on the shores of the Puget Sound, offering beautiful waterfront views and opportunities for water activities. It is also close to Lake Washington and the Cascade Range, providing ample opportunities for outdoor recreation, including hiking, skiing, and boating.\n",
      "\n",
      "4. Green City: Seattle is known for its commitment to sustainability and environmental consciousness. The city has a strong focus on recycling, renewable energy, and green initiatives. It has a well-developed public transportation system and is bike-friendly, making it easy to navigate without a car.\n",
      "\n",
      "5. Education and Research: Seattle is home to several prestigious universities and research institutions, including the University of Washington and Fred Hutchinson Cancer Research Center. These institutions contribute to the city's intellectual and scientific advancements and provide opportunities for education and research.\n",
      "\n",
      "6. Food and Coffee Culture: Seattle is renowned for its food and coffee culture. The city is known for its diverse culinary scene, offering a wide range of international cuisines and fresh seafood. Seattle is also the birthplace of Starbucks and has a thriving coffee culture, with numerous local coffee shops and roasters.\n",
      "\n",
      "These positive aspects make Seattle an attractive city to live in, work, and explore.\n",
      "\u001b[0m\u001b[1;3;38;2;237;90;200m[recursive_retriever] A: The sports teams in Boston include the Boston Red Sox, New England Patriots, Boston Celtics, Boston Bruins, and New England Revolution.\n",
      "\u001b[0m\u001b[1;3;38;2;90;149;237m[recursive_retriever] A: Seattle has a thriving economy, a vibrant cultural scene, stunning natural beauty, a commitment to sustainability, renowned educational and research institutions, and a diverse food and coffee culture. These positive aspects make Seattle an attractive city to live in, work, and explore.\n",
      "\u001b[0mThe sports teams in Boston include the Boston Red Sox, New England Patriots, Boston Celtics, Boston Bruins, and New England Revolution. Seattle, on the other hand, has a thriving economy, a vibrant cultural scene, stunning natural beauty, a commitment to sustainability, renowned educational and research institutions, and a diverse food and coffee culture. These positive aspects make Seattle an attractive city to live in, work, and explore.\n"
     ]
    }
   ],
   "source": [
    "response = query_engine.query(\n",
    "    \"Tell me about the sports teams in Boston and the positive aspects of Seattle\"\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OhpZWvuz_K2W"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0e93a2479384409a87ec321dafc37a27": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_31b34df9ba6a445d93cc6233e3f4b824",
       "IPY_MODEL_78e44903bbf34a338ebaf98b0ad7a090",
       "IPY_MODEL_c528a5c26e0a493d9cbadec7b61eb0e2"
      ],
      "layout": "IPY_MODEL_78c6176c32a047869fb0912271fcab75"
     }
    },
    "11413f3af9574736a5a52cd8f5582778": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ad697e475cb14f1c8c201209511d4e3a",
      "placeholder": "​",
      "style": "IPY_MODEL_8d5d68c4bde74c3bbf0ed61182728d0f",
      "value": " 17/17 [00:02&lt;00:00,  8.00it/s]"
     }
    },
    "128cc0fdcab64227a9f149e290f1a94f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1531d48a1a694db29793bd7db8596346": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_856d8052231246d9bf0c859379a8c11f",
      "max": 17,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2e54f4d63e0048dfbb2414c712d3fa0d",
      "value": 17
     }
    },
    "181218d0f5514e1a8346da804656f5d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1d28c3107cf04d42bd2c08c6f5b5c55f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2049d632a953467fb48f14ec63a88261": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "27439087326c40d8bf37a400fdfe6705": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2e54f4d63e0048dfbb2414c712d3fa0d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "31b34df9ba6a445d93cc6233e3f4b824": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6f74f396370b4e3e92f8634d20f59546",
      "placeholder": "​",
      "style": "IPY_MODEL_1d28c3107cf04d42bd2c08c6f5b5c55f",
      "value": "Upserted vectors: 100%"
     }
    },
    "6f74f396370b4e3e92f8634d20f59546": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "73d9bfd4f94f44ed9a472a101be84a0b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "78c6176c32a047869fb0912271fcab75": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "78e44903bbf34a338ebaf98b0ad7a090": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_870b5a6f264549ebae86c611abc4e5ed",
      "max": 23,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_181218d0f5514e1a8346da804656f5d0",
      "value": 23
     }
    },
    "798981bae72742e6bc59c5ed89a20d68": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7b41f3c50aa646af812e3c1660724942": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7c32e2ba7ace4d158695d6375aead04d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2049d632a953467fb48f14ec63a88261",
      "placeholder": "​",
      "style": "IPY_MODEL_7b41f3c50aa646af812e3c1660724942",
      "value": " 17/17 [00:01&lt;00:00, 11.00it/s]"
     }
    },
    "856d8052231246d9bf0c859379a8c11f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "870b5a6f264549ebae86c611abc4e5ed": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "89ece7effef745218c0ae340cb49459d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9dbbeac38c304324878830799c49f090",
       "IPY_MODEL_b288f9667a7d4240bab98b0390326cd9",
       "IPY_MODEL_7c32e2ba7ace4d158695d6375aead04d"
      ],
      "layout": "IPY_MODEL_128cc0fdcab64227a9f149e290f1a94f"
     }
    },
    "8d5d68c4bde74c3bbf0ed61182728d0f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9300a17d34154ffb87d819cc869afb6d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9a13a789d9e04d20b3e98343832ff7e0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9b198221deb4448cb433370898fbe243": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9cc2a6a3ed3649dbadd7e0ab8d45b573": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9dbbeac38c304324878830799c49f090": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9300a17d34154ffb87d819cc869afb6d",
      "placeholder": "​",
      "style": "IPY_MODEL_798981bae72742e6bc59c5ed89a20d68",
      "value": "Upserted vectors: 100%"
     }
    },
    "a956afde996449319b10ddb082bb8fc2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ddf3f76146984a8596ec8e734f9b2767",
       "IPY_MODEL_1531d48a1a694db29793bd7db8596346",
       "IPY_MODEL_11413f3af9574736a5a52cd8f5582778"
      ],
      "layout": "IPY_MODEL_73d9bfd4f94f44ed9a472a101be84a0b"
     }
    },
    "ad697e475cb14f1c8c201209511d4e3a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b288f9667a7d4240bab98b0390326cd9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d765e261007c4a618f8ed209ef8d97ed",
      "max": 17,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9cc2a6a3ed3649dbadd7e0ab8d45b573",
      "value": 17
     }
    },
    "c528a5c26e0a493d9cbadec7b61eb0e2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_27439087326c40d8bf37a400fdfe6705",
      "placeholder": "​",
      "style": "IPY_MODEL_f944441a44c944459a45e694c43d6c8b",
      "value": " 23/23 [00:01&lt;00:00, 13.15it/s]"
     }
    },
    "d765e261007c4a618f8ed209ef8d97ed": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ddf3f76146984a8596ec8e734f9b2767": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9b198221deb4448cb433370898fbe243",
      "placeholder": "​",
      "style": "IPY_MODEL_9a13a789d9e04d20b3e98343832ff7e0",
      "value": "Upserted vectors: 100%"
     }
    },
    "f944441a44c944459a45e694c43d6c8b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}