Skip to content

Commit

Permalink
Change to ingestor interface
Browse files Browse the repository at this point in the history
  • Loading branch information
@ChrisJar
ChrisJar committed Nov 14, 2024
1 parent aed6019 commit ed2c1a2
Show file tree
Hide file tree
Showing 3 changed files with 432 additions and 313 deletions.
103 changes: 27 additions & 76 deletions examples/langchain_multimodal_rag.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,20 @@
"This notebook shows how to perform RAG on the table, chart, and text extraction results of nv-ingest's pdf extraction tools using LangChain"
]
},
{
"cell_type": "markdown",
"id": "c6905d11-0ec3-43c8-961b-24cb52e36bfe",
"metadata": {},
"source": [
"**Note:** In order to run this notebook, you'll need to have the NV-Ingest microservice running along with all of the other included microservices. To do this, make sure all of the services are uncommented in the file: [docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml) and follow the [quickstart guide](https://github.com/NVIDIA/nv-ingest?tab=readme-ov-file#quickstart) to start everything up."
]
},
{
"cell_type": "markdown",
"id": "81014734-f765-48fc-8fc2-4c19f5f28eae",
"metadata": {},
"source": [
"To start we'll need to make sure we have Langchain installed as well as pymilvus so that we can connect to the Milvus vector database (VDB) that NV-Ingest uses to store embeddings"
"To start, make sure Langchain and pymilvus are installed and up to date"
]
},
{
Expand All @@ -39,7 +47,7 @@
"id": "d888ba26-04cf-4577-81a3-5bcd537fc2f6",
"metadata": {},
"source": [
"Then, we'll use NV-Ingest to parse an example pdf that contains text, tables, charts, and images, embed it with the included embedding microservice and store the results in the Milvus vector database. We'll need to make sure to have the NV-Ingest microservice up and running at localhost:7670 along with the supporting NIMs and microservices. To do this, follow the nv-ingest [quickstart guide](https://github.com/NVIDIA/nv-ingest?tab=readme-ov-file#quickstart). This notebook requires all of the services to be [running](https://github.com/NVIDIA/nv-ingest/blob/main/docs/deployment.md#launch-nv-ingest-micro-services). Once everything is ready, we can create a job with the NV-Ingest python client"
"Then, we'll use NV-Ingest's Ingestor interface extract the tables and charts from a test pdf, embed them, and upload them to our Milvus vector database (VDB)"
]
},
{
Expand All @@ -49,87 +57,30 @@
"metadata": {},
"outputs": [],
"source": [
"from nv_ingest_client.client import NvIngestClient\n",
"from nv_ingest_client.primitives import JobSpec\n",
"from nv_ingest_client.primitives.tasks import ExtractTask\n",
"from nv_ingest_client.primitives.tasks import EmbedTask\n",
"from nv_ingest_client.primitives.tasks import VdbUploadTask\n",
"\n",
"\n",
"from nv_ingest_client.util.file_processing.extract import extract_file_content\n",
"import logging, time\n",
"\n",
"logger = logging.getLogger(\"nv_ingest_client\")\n",
"\n",
"file_name = \"../data/multimodal_test.pdf\"\n",
"file_content, file_type = extract_file_content(file_name)\n",
"\n",
"client = NvIngestClient(\n",
" message_client_hostname=\"localhost\",\n",
" message_client_port=7670\n",
")\n",
"\n",
"job_spec = JobSpec(\n",
" document_type=file_type,\n",
" payload=file_content,\n",
" source_id=file_name,\n",
" source_name=file_name,\n",
" extended_options={\n",
" \"tracing_options\": {\n",
" \"trace\": True,\n",
" \"ts_send\": time.time_ns()\n",
" }\n",
" },\n",
")"
]
},
{
"cell_type": "markdown",
"id": "7fa987e6-8bb1-4ed1-97c2-540e136e2d19",
"metadata": {},
"source": [
"And then, we can add and submit tasks to extract the text, tables, and charts from the example pdf, generate embeddings from the results, and store them in the Milvus VDB"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8bcd6b2a-c832-4685-bd6e-53b60a107e28",
"metadata": {},
"outputs": [],
"source": [
"extract_task = ExtractTask(\n",
" document_type=file_type,\n",
" extract_text=True,\n",
" extract_images=False,\n",
" extract_tables=True,\n",
" extract_charts=True,\n",
")\n",
"\n",
"embed_task = EmbedTask(\n",
" text=True,\n",
" tables=True,\n",
"from nv_ingest_client.client import Ingestor\n",
"\n",
"ingestor = (\n",
" Ingestor(message_client_hostname=\"localhost\")\n",
" .files(\"../data/multimodal_test.pdf\")\n",
" .extract(\n",
" extract_text=False,\n",
" extract_tables=True,\n",
" extract_images=False,\n",
" ).embed(\n",
" text=False,\n",
" tables=True,\n",
" ).vdb_upload()\n",
")\n",
"\n",
"vdb_upload_task = VdbUploadTask()\n",
"\n",
"job_spec.add_task(extract_task)\n",
"job_spec.add_task(embed_task)\n",
"job_spec.add_task(vdb_upload_task)\n",
"\n",
"job_id = client.add_job(job_spec)\n",
"\n",
"client.submit_job(job_id, \"morpheus_task_queue\")\n",
"\n",
"result = client.fetch_job_result(job_id, timeout=60)"
"results = ingestor.ingest()"
]
},
{
"cell_type": "markdown",
"id": "02131711-31bf-4536-81b7-8c464c7473e3",
"metadata": {},
"source": [
"Now, the text, table, and chart content is extracted and stored in the Milvus VDB along with the embeddings. Next we'll connect LlamaIndex to Milvus and create a vector store so that we can query our extraction results. The vector store must use the same embedding model as the embedding service in NV-Ingest: `nv-embed-qa-e5-v5`"
"Now, the text, table, and chart content is extracted and stored in the Milvus VDB along with the embeddings. Next we'll connect LlamaIndex to Milvus and create a vector store so that we can query our extraction results"
]
},
{
Expand All @@ -142,10 +93,10 @@
"from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings\n",
"from langchain_milvus import Milvus\n",
"\n",
"embedding = NVIDIAEmbeddings(base_url=\"http://localhost:8012/v1\", model=\"nvidia/nv-embedqa-e5-v5\")\n",
"embedding_function = NVIDIAEmbeddings(base_url=\"http://localhost:8012/v1\")\n",
"\n",
"vectorstore = Milvus(\n",
" embedding_function=embedding,\n",
" embedding_function=embedding_function,\n",
" collection_name=\"nv_ingest_collection\",\n",
" primary_field = \"pk\",\n",
" vector_field = \"vector\",\n",
Expand Down
103 changes: 27 additions & 76 deletions examples/llama_index_multimodal_rag.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,118 +18,69 @@
},
{
"cell_type": "markdown",
"id": "baecfda5-137b-43da-a8d4-23dd47131be9",
"metadata": {},
"source": [
"To start we'll need to make sure we have LlamaIndex installed as well as pymilvus so that we can connect to the Milvus vector database (VDB) that NV-Ingest uses to store embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d75fdaa-3085-4b77-9289-15276d5cd1c5",
"id": "c65edc4b-2084-47c9-a837-733264201802",
"metadata": {},
"outputs": [],
"source": [
"pip install -qU llama_index llama-index-embeddings-nvidia llama-index-llms-nvidia llama-index-vector-stores-milvus pymilvus"
"**Note:** In order to run this notebook, you'll need to have the NV-Ingest microservice running along with all of the other included microservices. To do this, make sure all of the services are uncommented in the file: [docker-compose.yaml](https://github.com/NVIDIA/nv-ingest/blob/main/docker-compose.yaml) and follow the [quickstart guide](https://github.com/NVIDIA/nv-ingest?tab=readme-ov-file#quickstart) to start everything up."
]
},
{
"cell_type": "markdown",
"id": "45412661-9516-47f9-8bea-6f857e0e173f",
"id": "baecfda5-137b-43da-a8d4-23dd47131be9",
"metadata": {},
"source": [
"Then, we'll use NV-Ingest to parse an example pdf that contains text, tables, charts, and images, embed it with the included embedding microservice and store the results in the Milvus vector database. We'll need to make sure to have the NV-Ingest microservice up and running at localhost:7670 along with the supporting NIMs and microservices. To do this, follow the nv-ingest [quickstart guide](https://github.com/NVIDIA/nv-ingest?tab=readme-ov-file#quickstart). This notebook requires all of the services to be [running](https://github.com/NVIDIA/nv-ingest/blob/main/docs/deployment.md#launch-nv-ingest-micro-services). Once everything is ready, we can create a job with the NV-Ingest python client"
"To start make sure that LlamaIndex and pymilvus are installed and up to date"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9337daf9-7342-427a-a95e-2d9ac9b5076b",
"execution_count": null,
"id": "3d75fdaa-3085-4b77-9289-15276d5cd1c5",
"metadata": {},
"outputs": [],
"source": [
"from nv_ingest_client.client import NvIngestClient\n",
"from nv_ingest_client.primitives import JobSpec\n",
"from nv_ingest_client.primitives.tasks import ExtractTask\n",
"from nv_ingest_client.primitives.tasks import EmbedTask\n",
"from nv_ingest_client.primitives.tasks import VdbUploadTask\n",
"\n",
"\n",
"from nv_ingest_client.util.file_processing.extract import extract_file_content\n",
"import logging, time\n",
"\n",
"logger = logging.getLogger(\"nv_ingest_client\")\n",
"\n",
"file_name = \"../data/multimodal_test.pdf\"\n",
"file_content, file_type = extract_file_content(file_name)\n",
"\n",
"client = NvIngestClient(\n",
" message_client_hostname=\"localhost\",\n",
" message_client_port=7670\n",
")\n",
"\n",
"job_spec = JobSpec(\n",
" document_type=file_type,\n",
" payload=file_content,\n",
" source_id=file_name,\n",
" source_name=file_name,\n",
" extended_options={\n",
" \"tracing_options\": {\n",
" \"trace\": True,\n",
" \"ts_send\": time.time_ns()\n",
" }\n",
" },\n",
")"
"pip install -qU llama_index llama-index-embeddings-nvidia llama-index-llms-nvidia llama-index-vector-stores-milvus pymilvus"
]
},
{
"cell_type": "markdown",
"id": "2aa9a74c-f7e4-475d-970d-cf820cd8ea19",
"id": "45412661-9516-47f9-8bea-6f857e0e173f",
"metadata": {},
"source": [
"And then, we can add and submit tasks to extract the text, tables, and charts from the example pdf, generate embeddings from the results, and store them in the Milvus VDB"
"Then, we'll use NV-Ingest's Ingestor interface extract the tables and charts from a test pdf, embed them, and upload them to our Milvus vector database (VDB)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6f07d820-0ae6-4681-88da-3f8857ea71fb",
"execution_count": 5,
"id": "9337daf9-7342-427a-a95e-2d9ac9b5076b",
"metadata": {},
"outputs": [],
"source": [
"extract_task = ExtractTask(\n",
" document_type=file_type,\n",
" extract_text=True,\n",
" extract_images=False,\n",
" extract_tables=True,\n",
" extract_charts=True,\n",
")\n",
"\n",
"embed_task = EmbedTask(\n",
" text=True,\n",
" tables=True,\n",
"from nv_ingest_client.client import Ingestor\n",
"\n",
"ingestor = (\n",
" Ingestor(message_client_hostname=\"localhost\")\n",
" .files(\"../data/multimodal_test.pdf\")\n",
" .extract(\n",
" extract_text=False,\n",
" extract_tables=True,\n",
" extract_images=False,\n",
" ).embed(\n",
" text=False,\n",
" tables=True,\n",
" ).vdb_upload()\n",
")\n",
"\n",
"vdb_upload_task = VdbUploadTask()\n",
"\n",
"job_spec.add_task(extract_task)\n",
"job_spec.add_task(embed_task)\n",
"job_spec.add_task(vdb_upload_task)\n",
"\n",
"job_id = client.add_job(job_spec)\n",
"\n",
"client.submit_job(job_id, \"morpheus_task_queue\")\n",
"\n",
"result = client.fetch_job_result(job_id, timeout=60)"
"results = ingestor.ingest()"
]
},
{
"cell_type": "markdown",
"id": "eaf6a9b8-83cf-4061-bf57-d50edc3978d0",
"metadata": {},
"source": [
"Now, the text, table, and chart content is extracted and stored in the Milvus VDB along with the embeddings. Next, we'll connect LlamaIndex to Milvus and create a vector store index so that we can query our extraction results. The vector store index must use the same embedding model as the embedding service in NV-Ingest: `nv-embed-qa-e5-v5`"
"Now, the text, table, and chart content is extracted and stored in the Milvus VDB along with the embeddings. Next, we'll connect LlamaIndex to Milvus and create a vector store index so that we can query our extraction results"
]
},
{
Expand All @@ -143,7 +94,7 @@
"from llama_index.embeddings.nvidia import NVIDIAEmbedding\n",
"from llama_index.vector_stores.milvus import MilvusVectorStore\n",
"\n",
"embed_model = NVIDIAEmbedding(base_url=\"http://localhost:8012/v1\", model=\"nvidia/nv-embedqa-e5-v5\")\n",
"embed_model = NVIDIAEmbedding(base_url=\"http://localhost:8012/v1\")\n",
"\n",
"vector_store = MilvusVectorStore(\n",
" uri=\"http://localhost:19530\",\n",
Expand Down
Loading

0 comments on commit ed2c1a2

Please sign in to comment.