Contains scripts that integrates LLM and Vector DB (ChromaDB) to test Retrieval Augmented Generation use cases. Two commonly used open-source frameworks that can be potential candidates to integrate with llama-cpp/llama-cpp-python:
CLI based RAG application for LangChain and LlamaIndex
Preduild Docker image (Image is based on Ampere Optimized PyTorch and llama-cpp):
# docker pull ghcr.io/amperecomputingai/local-rag:v0.0.1
# docker run -it --rm ghcr.io/amperecomputingai/local-rag:v0.0.1
First step is to get the llama model
wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf
Running LangChain RAG Application
# python langchain-cli.py
Welcome to llama CLI. Reserved first words help, upload and quit
Type "help" for available commands.
llama QA >> Who is president of India?
The current President of India is Ram Nath Kovind.
llama QA >> upload news_india.txt
Uploading news_india.txt [Done]
llama QA >> Who is president of India?
Droupadi Murmu
llama QA >> quit
At the time of "llama-2-7b-chat.Q4_K_M.gguf" training, president of India was "Ram Nath Kovind" With "upload news_india.txt", latest news regarging president of India is uploaded to RAG. Asking same question again provides the correct answer "Droupadi Murmu"
Running LlamaIndex RAG Application
# python llamaindex-cli.py
Welcome to llama CLI. Reserved first words help, upload and quit
Type "help" for available commands.
llama QA >> who is president of USA?
president of the United States is Joe Biden.
llama QA >> upload news_usa.txt
Uploading news_usa.txt [Done]
llama QA >> who is president of USA?
answer to the query is "Tom Cruise".
llama QA >> quit
Fake news (news_usa.txt) is uploaded to RAG. Which changes the answer from "Joe Biden" to "Tom Cruise"
For best results use Ampere Optimized local-rag Docker image on OCI A1 instance.
System requirement: 64 OCPUs + 128GB Memory.