cookbook/Using_Nemo_Guardrails_with_LiteLLM_Server.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eKXncoQbU_2j"
   },
   "source": [
    "# Using Nemo-Guardrails with LiteLLM Server\n",
    "\n",
    "[Call Bedrock, TogetherAI, Huggingface, etc. on the server](https://docs.litellm.ai/docs/providers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZciYaLwvuFbu"
   },
   "source": [
    "## Using with Bedrock\n",
    "\n",
    "`docker run -e PORT=8000 -e AWS_ACCESS_KEY_ID=<your-aws-access-key> -e AWS_SECRET_ACCESS_KEY=<your-aws-secret-key> -p 8000:8000 ghcr.io/berriai/litellm:latest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vOUwGSJ2Vsy3"
   },
   "outputs": [],
   "source": [
    "pip install nemoguardrails langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xXEJNxe7U0IN"
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model_name=\"anthropic.claude-v2\", openai_api_base=\"http://0.0.0.0:8000\", openai_api_key=\"my-fake-key\")\n",
    "\n",
    "from nemoguardrails import LLMRails, RailsConfig\n",
    "\n",
    "config = RailsConfig.from_path(\"./config.yml\")\n",
    "app = LLMRails(config, llm=llm)\n",
    "\n",
    "new_message = app.generate(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": \"Hello! What can you do for me?\"\n",
    "}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vz5n00qyuKjp"
   },
   "source": [
    "## Using with TogetherAI\n",
    "\n",
    "1. You can either set this in the server environment:\n",
    "`docker run -e PORT=8000 -e TOGETHERAI_API_KEY=<your-together-ai-api-key> -p 8000:8000 ghcr.io/berriai/litellm:latest`\n",
    "\n",
    "2. **Or** Pass this in as the api key `(...openai_api_key=\"<your-together-ai-api-key>\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XK1sk-McuhpE"
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model_name=\"together_ai/togethercomputer/CodeLlama-13b-Instruct\", openai_api_base=\"http://0.0.0.0:8000\", openai_api_key=\"my-together-ai-api-key\")\n",
    "\n",
    "from nemoguardrails import LLMRails, RailsConfig\n",
    "\n",
    "config = RailsConfig.from_path(\"./config.yml\")\n",
    "app = LLMRails(config, llm=llm)\n",
    "\n",
    "new_message = app.generate(messages=[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": \"Hello! What can you do for me?\"\n",
    "}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8A1KWKnzuxAS"
   },
   "source": [
    "### CONFIG.YML\n",
    "\n",
    "save this example `config.yml` in your current directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NKN1GmSvu0Cx"
   },
   "outputs": [],
   "source": [
    "# instructions:\n",
    "#   - type: general\n",
    "#     content: |\n",
    "#       Below is a conversation between a bot and a user about the recent job reports.\n",
    "#       The bot is factual and concise. If the bot does not know the answer to a\n",
    "#       question, it truthfully says it does not know.\n",
    "\n",
    "# sample_conversation: |\n",
    "#   user \"Hello there!\"\n",
    "#     express greeting\n",
    "#   bot express greeting\n",
    "#     \"Hello! How can I assist you today?\"\n",
    "#   user \"What can you do for me?\"\n",
    "#     ask about capabilities\n",
    "#   bot respond about capabilities\n",
    "#     \"I am an AI assistant that helps answer mathematical questions. My core mathematical skills are powered by wolfram alpha.\"\n",
    "#   user \"What's 2+2?\"\n",
    "#     ask math question\n",
    "#   bot responds to math question\n",
    "#     \"2+2 is equal to 4.\"\n",
    "\n",
    "# models:\n",
    "#   - type: main\n",
    "#     engine: openai\n",
    "#     model: claude-instant-1"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}