1_get_chunks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieval API DEMO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Initialisation\n",
    "Dependencies and environment initialisation. Ensure there's a `.env` file with your credentials in the same directory as this script. Use the `env` file as reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "import requests as r\n",
    "from IPython.display import Markdown\n",
    "import utils as u\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_HOST = 'api.dowjones.com'\n",
    "AUTH_HOST = 'accounts.dowjones.com'\n",
    "CLIENT_ID = os.getenv('FACTIVA_CLIENTID')\n",
    "USERNAME = os.getenv('FACTIVA_USERNAME')\n",
    "PASSWORD = os.getenv('FACTIVA_PASSWORD')\n",
    "AUTH_URL = f\"https://{AUTH_HOST}/oauth2/v1/token\"\n",
    "GCP_PROJECT = os.getenv('GCP_PROJECT')\n",
    "GCP_LOCATION = os.getenv('GCP_LOCATION')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication - Generate Bearer\n",
    "\n",
    "For details about getting the `bearer_token`, please see the `utils.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Authentication Successful**: Bearer token OK for user 9ZZZ159100-svcaccount"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bearer_token = u.get_bearer_token(CLIENT_ID, USERNAME, PASSWORD, AUTH_URL)\n",
    "if bearer_token:\n",
    "    display(Markdown(f\"**Authentication Successful**: Bearer token OK for user {USERNAME.split('@')[0]}\"))\n",
    "else:\n",
    "    display(Markdown('**Authentication Failed**: Cannot obtain the Bearer token'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "req_headers = {\n",
    "    \"Authorization\": f\"Bearer {bearer_token}\",\n",
    "    \"Content-Type\": \"application/json\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factiva Retrieval API Query\n",
    "\n",
    "### Prompt\n",
    "Add the prompt to be sent to the Retrieval API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"What are NASA's planned missions to the Moon in 2025, and what are their primary objectives?\"\n",
    "# \"中国2025年计划进行哪些月球探测任务？其主要目标是什么？\"   # What are China's planned missions to the Moon in 2025, and what are their primary objectives?\n",
    "# \"今後12か月間の日本の半導体業界の見通しはどのようなものですか?\"  # What are the perspectives for the semiconductors industry in Japan for the next 12 months?\n",
    "# \"Summarise the latest earnings report from Microsoft Corp\"\n",
    "# \"What are the perspectives of bitcoin for the next 12 months?\"\n",
    "frapi_prompt = \"What are NASA's planned missions to the Moon in 2025, and what are their primary objectives?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the Retrieval API payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "frapi_query = {\n",
    "  \"data\": {\n",
    "    \"attributes\": {\n",
    "      \"response_limit\": 5,\n",
    "      \"query\": {\n",
    "        \"search_filters\": [\n",
    "          {\n",
    "            \"scope\": \"Language\",\n",
    "            \"value\": \"en\"       # en or ja or de or it or es\n",
    "          }\n",
    "        ],\n",
    "\n",
    "        \"value\": frapi_prompt\n",
    "      }\n",
    "    },\n",
    "    \"id\": \"GenAIRetrievalExample\",\n",
    "    \"type\": \"genai-content\"\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Send Query and Receive Chunks from the Retrieval API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully retrieved chunks\n"
     ]
    }
   ],
   "source": [
    "chunks_resp = r.post(f\"https://{API_HOST}/content/gen-ai/retrieve\", json=frapi_query, headers=req_headers)\n",
    "\n",
    "if chunks_resp.status_code == 200:\n",
    "    print('Successfully retrieved chunks')\n",
    "else:\n",
    "    print(f\"Request Failed: {chunks_resp.json()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "###                            NASA revealed the missions it will carry out during 2025"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**CE NoticiasFinancieras** - 2025-01-27 - NFINCE0020250127el1r004n7 - Lang: en"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "The directors of the North American Aerospace Agency (NASA) reported that in 2025 it will focus on consolidating its leadership in space exploration t..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### The most anticipated space missions of 2025"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**CE NoticiasFinancieras** - 2024-12-31 - NFINCE0020250101ekcv000n2 - Lang: en"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "In the coming months, different space agencies and private companies will launch several missions that promise to improve our understanding of the uni..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### What are the most important missions to be carried out by NASA in 2025?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**CE NoticiasFinancieras** - 2025-01-04 - NFINCE0020250104el14009ja - Lang: en"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "In 2025, NASA will focus on consolidating its leadership in space exploration through ambitious missions ranging from studying deep space to strengthe..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### Find out what the space missions are for 2025"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**CE NoticiasFinancieras** - 2025-01-04 - NFINCE0020250104el1400abk - Lang: en"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Multiple countries such as the United States, China, Japan and India are planning new milestones in global astronomy. NASA and commercial space compan..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "### From new commercial Moon landers to asteroid investigations, expect a slate of exciting space missions in 2025"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**The Conversation** - 2024-12-26 - CONVAU0020241226ekcq00007 - Lang: en"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Zhenbo Wang, University of Tennessee From new commercial Moon landers to asteroid investigations, expect a slate of exciting space missions in 2025 Zh..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "chunks_list = chunks_resp.json()['data']\n",
    "\n",
    "# u.print_full_chunks(chunks_list)\n",
    "u.print_partial_chunks(chunks_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval API Conclusion\n",
    "\n",
    "Up to this point it the seen functionalities are connected with the Retrieval API.\n",
    "\n",
    "However, given that this service is an intermediate component in a full-stack solution, the below Test LLM steps and the [Read Article](2_read_article.ipynb) notebook are the two complementary avenues to get a full working solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a downstream step that is only illustrative on how the response generation stage can be implemented. In the below example only a few articles are used to respond the prompt using an LLM hosted in **Google Cloud Vertex AI**. The passed request is built from the original prompt enhanced with the retrieved articles as gounding context.\n",
    "\n",
    "The tested LLM is Google Gemini 2.0 Flash. Response generations took a few seconds for a context between 3K to 4K LLM tokens.\n",
    "\n",
    "### Prerequisite\n",
    "\n",
    "To successfully complete the following steps, it's important that there's an active GCloud authentication in the environment where this notbeook is executed from. This can be done by running the following command:\n",
    "\n",
    "```bash\n",
    "$ gcloud auth application-default login\n",
    "```\n",
    "\n",
    "For more information see [Set up ADC for a local development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).\n",
    "\n",
    "### Gemini Structured Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "instructions_text =\"\"\"\n",
    "    You are an experienced business analyst that respond in a professional manner.\n",
    "    Answer the query using only the information provided in the list of articles.\n",
    "    If you use information from an article, cite it using squared brackets containing the index number.\n",
    "    At the end of the answer, show a list of the cited articles ordered by index and under the title Cited Articles.\n",
    "    Each cited article must be displayed in the following Markdown format:\n",
    "    - [index] [headline - source_name - publication_date](url)\n",
    "    Use Markdown for the output.\n",
    "\"\"\"\n",
    "article_list = []\n",
    "\n",
    "for chunk in chunks_list:\n",
    "    article = {\n",
    "        'index': len(article_list) + 1,\n",
    "        'url': f\"https://dj.factiva.com/article?id=drn:archive.newsarticle.{str(chunk['meta']['original_doc_id']).strip()}\",\n",
    "        'source_name': str(chunk['meta']['source']['name']).strip(),\n",
    "        'headline': str(chunk['attributes']['headline']['main']['text']).strip(),\n",
    "        'publication_date': chunk['attributes']['publication_date'],\n",
    "        'content': f\"{str(chunk['attributes']['snippet']['content'][0]['text']).strip()} {str(chunk['attributes']['content'][0]['text']).strip()}\"\n",
    "    }\n",
    "    article_list.append(article)\n",
    "\n",
    "llm_prompt = {\n",
    "    'query': frapi_prompt,\n",
    "    'articles': article_list,\n",
    "    \"instructions\": instructions_text.strip()\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"query\": \"What are NASA's planned missions to the Moon in 2025, and what are their primary objectives?\",\n",
      "    \"articles\": [\n",
      "        {\n",
      "            \"index\": 1,\n",
      "            \"url\": \"https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250127el1r004n7\",\n",
      "            \"source_name\": \"CE NoticiasFinancieras\",\n",
      "            \"headline\": \"NASA revealed the missions it will carry out during 2025\",\n",
      "            \"publication_date\": \"2025-01-27\",\n",
      "            \"content\": \"The directors of the North American Aerospace Agency (NASA) reported that in 2025 it will focus on consolidating its leadership in space exploration t...\"\n",
      "        },\n",
      "        {\n",
      "            \"index\": 2,\n",
      "            \"url\": \"https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250101ekcv000n2\",\n",
      "            \"source_name\": \"CE NoticiasFinancieras\",\n",
      "            \"headline\": \"The most anticipated space missions of 2025\",\n",
      "            \"publication_date\": \"2024-12-31\",\n",
      "            \"content\": \"In the coming months, different space agencies and private companies will launch several missions that promise to improve our understanding of the uni...\"\n",
      "        },\n",
      "        {\n",
      "            \"index\": 3,\n",
      "            \"url\": \"https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250104el14009ja\",\n",
      "            \"source_name\": \"CE NoticiasFinancieras\",\n",
      "            \"headline\": \"What are the most important missions to be carried out by NASA in 2025?\",\n",
      "            \"publication_date\": \"2025-01-04\",\n",
      "            \"content\": \"In 2025, NASA will focus on consolidating its leadership in space exploration through ambitious missions ranging from studying deep space to strengthe...\"\n",
      "        },\n",
      "        {\n",
      "            \"index\": 4,\n",
      "            \"url\": \"https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250104el1400abk\",\n",
      "            \"source_name\": \"CE NoticiasFinancieras\",\n",
      "            \"headline\": \"Find out what the space missions are for 2025\",\n",
      "            \"publication_date\": \"2025-01-04\",\n",
      "            \"content\": \"Multiple countries such as the United States, China, Japan and India are planning new milestones in global astronomy. NASA and commercial space compan...\"\n",
      "        },\n",
      "        {\n",
      "            \"index\": 5,\n",
      "            \"url\": \"https://dj.factiva.com/article?id=drn:archive.newsarticle.CONVAU0020241226ekcq00007\",\n",
      "            \"source_name\": \"The Conversation\",\n",
      "            \"headline\": \"From new commercial Moon landers to asteroid investigations, expect a slate of exciting space missions in 2025\",\n",
      "            \"publication_date\": \"2024-12-26\",\n",
      "            \"content\": \"Zhenbo Wang, University of Tennessee From new commercial Moon landers to asteroid investigations, expect a slate of exciting space missions in 2025 Zh...\"\n",
      "        }\n",
      "    ],\n",
      "    \"instructions\": \"You are an experienced business analyst that respond in a professional manner.\\n    Answer the query using only the information provided in the list of articles.\\n    If you use information from an article, cite it using squared brackets containing the index number.\\n    At the end of the answer, show a list of the cited articles ordered by index and under the title Cited Articles.\\n    Each cited article must be displayed in the following Markdown format:\\n    - [index] [headline - source_name - publication_date](url)\\n    Use Markdown for the output.\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# u.print_full_llm_prompt(llm_prompt)\n",
    "u.print_partial_llm_prompt(llm_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gemini Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Prompt:** What are NASA's planned missions to the Moon in 2025, and what are their primary objectives? \n",
       " --------------------"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "In 2025, NASA has several planned missions to the Moon with the primary objective of laying the groundwork for a sustainable human presence and future exploration [1, 3]. These missions include:\n",
       "\n",
       "*   **IM-2 Mission:** Scheduled by Intuitive Machines to land at the lunar south pole, this mission will drill into the lunar subsurface and measure volatiles using an advanced drill and mass spectrometer [1, 3]. This is a crucial test for future Artemis manned missions [1, 3]. The spacecraft will also carry a small satellite to map water deposits on the Moon to help NASA identify future landing sites for the Artemis mission [2].\n",
       "*   **Commercial Lunar Payload Services (CLPS) program:** This program will continue to play a central role in developing technologies and scientific experiments to facilitate a sustainable human presence on the Moon [1, 3, 5]. Missions include deliveries by companies such as Astrobotic, Intuitive Machines, and Firefly Aerospace [5].\n",
       "*   **Blue Ghost Mission:** Developed by Firefly Aerospace, this mission will spend two weeks on the Moon as part of NASA's Commercial Lunar Cargo Services [4]. It aims to carry a vehicle with 10 payloads from NASA and will head for Mons Latreille, gathering information about the lunar surface and how lunar rocks interact with the solar wind and Earth's magnetic field [2].\n",
       "*   **M2/Resilience mission:** Japan’s mission will launch a lander and micro-rover to the lunar surface to study the lunar soil to understand its composition and properties. Researchers will also conduct a water-splitting test to produce oxygen and hydrogen by extracting water from the lunar surface [5].\n",
       "\n",
       "These missions are critical to the success of the Artemis program and aim to expand human knowledge and consolidate strategic alliances with international and private partners [1, 3].\n",
       "\n",
       "Cited Articles:\n",
       "\n",
       "*   [1] [NASA revealed the missions it will carry out during 2025 - CE NoticiasFinancieras - 2025-01-27](https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250127el1r004n7)\n",
       "*   [2] [The most anticipated space missions of 2025 - CE NoticiasFinancieras - 2024-12-31](https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250101ekcv000n2)\n",
       "*   [3] [What are the most important missions to be carried out by NASA in 2025? - CE NoticiasFinancieras - 2025-01-04](https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250104el14009ja)\n",
       "*   [4] [Find out what the space missions are for 2025 - CE NoticiasFinancieras - 2025-01-04](https://dj.factiva.com/article?id=drn:archive.newsarticle.NFINCE0020250104el1400abk)\n",
       "*   [5] [From new commercial Moon landers to asteroid investigations, expect a slate of exciting space missions in 2025 - The Conversation - 2024-12-26](https://dj.factiva.com/article?id=drn:archive.newsarticle.CONVAU0020241226ekcq00007)\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = u.gemini_generate(llm_prompt, GCP_PROJECT, GCP_LOCATION)\n",
    "display(Markdown(f\"**Prompt:** {frapi_prompt} \\n --------------------\"))\n",
    "display(Markdown(response))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fadev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}