LlamaIndex_Phi-2-FP16-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phi-2 (FP16)\n",
    "#### RAG with LlamaIndex - Nvidia CUDA + WSL (Windows Subsystem for Linux) + Word documents + Local LLM\n",
    "\n",
    "### Note: Uses FP16 model (not quantized) - possible issue with llamaCPP and the quantized versions. I was sometimes not getting a response (even to the same question)\n",
    "\n",
    "This notebook demonstrates the use of LlamaIndex for Retrieval Augmented Generation using Windows' WSL and an Nvidia's CUDA.\n",
    "\n",
    "See the [README.md](README.md) file for help on how to run this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Llama Index for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/markwsl/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load the Word document(s)\n",
    "\n",
    "Note: A fictitious story about Thundertooth a dinosaur who has travelled to the future. Thanks ChatGPT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 18 key-value pairs and 325 tensors from ./Models/phi-2-ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = phi2\n",
      "llama_model_loader: - kv   1:                               general.name str              = Phi2\n",
      "llama_model_loader: - kv   2:                        phi2.context_length u32              = 2048\n",
      "llama_model_loader: - kv   3:                      phi2.embedding_length u32              = 2560\n",
      "llama_model_loader: - kv   4:                   phi2.feed_forward_length u32              = 10240\n",
      "llama_model_loader: - kv   5:                           phi2.block_count u32              = 32\n",
      "llama_model_loader: - kv   6:                  phi2.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   7:               phi2.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   8:          phi2.attention.layer_norm_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv   9:                  phi2.rope.dimension_count u32              = 32\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,51200]   = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\n",
      "llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,51200]   = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,50000]   = [\"Ġ t\", \"Ġ a\", \"h e\", \"i n\", \"r e\",...\n",
      "llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 50256\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 50256\n",
      "llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 50256\n",
      "llama_model_loader: - type  f32:  195 tensors\n",
      "llama_model_loader: - type  f16:  130 tensors\n",
      "llm_load_vocab: mismatch in special tokens definition ( 910/51200 vs 944/51200 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = phi2\n",
      "llm_load_print_meta: vocab type       = BPE\n",
      "llm_load_print_meta: n_vocab          = 51200\n",
      "llm_load_print_meta: n_merges         = 50000\n",
      "llm_load_print_meta: n_ctx_train      = 2048\n",
      "llm_load_print_meta: n_embd           = 2560\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 32\n",
      "llm_load_print_meta: n_embd_head_k    = 80\n",
      "llm_load_print_meta: n_embd_head_v    = 80\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: n_embd_k_gqa     = 2560\n",
      "llm_load_print_meta: n_embd_v_gqa     = 2560\n",
      "llm_load_print_meta: f_norm_eps       = 1.0e-05\n",
      "llm_load_print_meta: f_norm_rms_eps   = 0.0e+00\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 10240\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 2048\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 3B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 2.78 B\n",
      "llm_load_print_meta: model size       = 5.18 GiB (16.01 BPW) \n",
      "llm_load_print_meta: general.name     = Phi2\n",
      "llm_load_print_meta: BOS token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: EOS token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: UNK token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: LF token         = 128 'Ä'\n",
      "llm_load_tensors: ggml ctx size =    0.25 MiB\n",
      "llm_load_tensors: offloading 30 repeating layers to GPU\n",
      "llm_load_tensors: offloaded 30/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =  5003.46 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  4503.22 MiB\n",
      ".............................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 2048\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:  CUDA_Host KV buffer size =    40.00 MiB\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =   600.00 MiB\n",
      "llama_new_context_with_model: KV self size  =  640.00 MiB, K (f16):  320.00 MiB, V (f16):  320.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    10.02 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   167.01 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =   168.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 5\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.ggml.unknown_token_id': '50256', 'tokenizer.ggml.eos_token_id': '50256', 'general.architecture': 'phi2', 'general.name': 'Phi2', 'phi2.context_length': '2048', 'tokenizer.ggml.model': 'gpt2', 'phi2.embedding_length': '2560', 'phi2.attention.head_count': '32', 'phi2.attention.head_count_kv': '32', 'phi2.feed_forward_length': '10240', 'phi2.attention.layer_norm_epsilon': '0.000010', 'phi2.block_count': '32', 'phi2.rope.dimension_count': '32', 'tokenizer.ggml.bos_token_id': '50256', 'general.file_type': '1'}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/phi-2-ggml-model-f16.gguf', # FP16 version of Phi-2\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=768,\n",
    "    context_window=2048, # Phi-2 2K context window - this could be a limitation for RAG as it has to put the content into this context window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 30}, # This is small model and there's no indication of layers offloaded to the GPU\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checkpoint\n",
    "\n",
    "Are you running on GPU? The above output should include near the top something like:\n",
    "> ggml_init_cublas: found 1 CUDA devices:\n",
    "\n",
    "And in the full text near the bottom should be:\n",
    "> llm_load_tensors: using CUDA for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Embeddings\n",
    "\n",
    "Convert your source document text into embeddings.\n",
    "\n",
    "The embedding model is from huggingface, this one performs well.\n",
    "\n",
    "> https://huggingface.co/thenlper/gte-large\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Prompt Template\n",
    "\n",
    "Prompt template for Phi-2 is below. As there's only a prompt we will combine the system message and prompt into the prompt.\n",
    "\n",
    "Instruct: {prompt}<br>\n",
    "Output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Produces a prompt specific to the model\n",
    "def modelspecific_prompt(promptmessage):\n",
    "    # As per https://huggingface.co/TheBloke/phi-2-GGUF\n",
    "    return f\"Instruct: {promptmessage}\\nOutput:\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Service Context\n",
    "\n",
    "For chunking the document into tokens using the embedding model and our LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size=128 # Number of tokens in each chunk, reduced for Phi-2 model\n",
    "Settings.chunk_overlap=20,\n",
    "Settings.context_window=2048, # This should be automatically set with the model metadata but we'll force it to ensure wit is\n",
    "Settings.num_output=768, # Maximum output from the LLM, let's put this at 512 to ensure LlamaIndex saves that \"space\" for the output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Index documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'<=' not supported between instances of 'int' and 'tuple'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[7], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m index \u001b[38;5;241m=\u001b[39m \u001b[43mVectorStoreIndex\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_documents\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdocuments\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/indices/base.py:135\u001b[0m, in \u001b[0;36mBaseIndex.from_documents\u001b[0;34m(cls, documents, storage_context, show_progress, callback_manager, transformations, service_context, **kwargs)\u001b[0m\n\u001b[1;32m    132\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m documents:\n\u001b[1;32m    133\u001b[0m     docstore\u001b[38;5;241m.\u001b[39mset_document_hash(doc\u001b[38;5;241m.\u001b[39mget_doc_id(), doc\u001b[38;5;241m.\u001b[39mhash)\n\u001b[0;32m--> 135\u001b[0m nodes \u001b[38;5;241m=\u001b[39m \u001b[43mrun_transformations\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    136\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdocuments\u001b[49m\u001b[43m,\u001b[49m\u001b[43m  \u001b[49m\u001b[38;5;66;43;03m# type: ignore\u001b[39;49;00m\n\u001b[1;32m    137\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtransformations\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    138\u001b[0m \u001b[43m    \u001b[49m\u001b[43mshow_progress\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mshow_progress\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    139\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    140\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    142\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mcls\u001b[39m(\n\u001b[1;32m    143\u001b[0m     nodes\u001b[38;5;241m=\u001b[39mnodes,\n\u001b[1;32m    144\u001b[0m     storage_context\u001b[38;5;241m=\u001b[39mstorage_context,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    149\u001b[0m     \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[1;32m    150\u001b[0m )\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/ingestion/pipeline.py:127\u001b[0m, in \u001b[0;36mrun_transformations\u001b[0;34m(nodes, transformations, in_place, cache, cache_collection, **kwargs)\u001b[0m\n\u001b[1;32m    125\u001b[0m             cache\u001b[38;5;241m.\u001b[39mput(\u001b[38;5;28mhash\u001b[39m, nodes, collection\u001b[38;5;241m=\u001b[39mcache_collection)\n\u001b[1;32m    126\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 127\u001b[0m         nodes \u001b[38;5;241m=\u001b[39m \u001b[43mtransform\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnodes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    129\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m nodes\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/interface.py:115\u001b[0m, in \u001b[0;36mNodeParser.__call__\u001b[0;34m(self, nodes, **kwargs)\u001b[0m\n\u001b[1;32m    114\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__call__\u001b[39m(\u001b[38;5;28mself\u001b[39m, nodes: List[BaseNode], \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m List[BaseNode]:\n\u001b[0;32m--> 115\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_nodes_from_documents\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnodes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/interface.py:75\u001b[0m, in \u001b[0;36mNodeParser.get_nodes_from_documents\u001b[0;34m(self, documents, show_progress, **kwargs)\u001b[0m\n\u001b[1;32m     70\u001b[0m doc_id_to_document \u001b[38;5;241m=\u001b[39m {doc\u001b[38;5;241m.\u001b[39mid_: doc \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m documents}\n\u001b[1;32m     72\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mevent(\n\u001b[1;32m     73\u001b[0m     CBEventType\u001b[38;5;241m.\u001b[39mNODE_PARSING, payload\u001b[38;5;241m=\u001b[39m{EventPayload\u001b[38;5;241m.\u001b[39mDOCUMENTS: documents}\n\u001b[1;32m     74\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m event:\n\u001b[0;32m---> 75\u001b[0m     nodes \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_parse_nodes\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdocuments\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mshow_progress\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mshow_progress\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     77\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m i, node \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(nodes):\n\u001b[1;32m     78\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m (\n\u001b[1;32m     79\u001b[0m             node\u001b[38;5;241m.\u001b[39mref_doc_id \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m     80\u001b[0m             \u001b[38;5;129;01mand\u001b[39;00m node\u001b[38;5;241m.\u001b[39mref_doc_id \u001b[38;5;129;01min\u001b[39;00m doc_id_to_document\n\u001b[1;32m     81\u001b[0m         ):\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/interface.py:179\u001b[0m, in \u001b[0;36mMetadataAwareTextSplitter._parse_nodes\u001b[0;34m(self, nodes, show_progress, **kwargs)\u001b[0m\n\u001b[1;32m    177\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m node \u001b[38;5;129;01min\u001b[39;00m nodes_with_progress:\n\u001b[1;32m    178\u001b[0m     metadata_str \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_metadata_str(node)\n\u001b[0;32m--> 179\u001b[0m     splits \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msplit_text_metadata_aware\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    180\u001b[0m \u001b[43m        \u001b[49m\u001b[43mnode\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_content\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmetadata_mode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mMetadataMode\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mNONE\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    181\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmetadata_str\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetadata_str\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    182\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    183\u001b[0m     all_nodes\u001b[38;5;241m.\u001b[39mextend(\n\u001b[1;32m    184\u001b[0m         build_nodes_from_splits(splits, node, id_func\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mid_func)\n\u001b[1;32m    185\u001b[0m     )\n\u001b[1;32m    187\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m all_nodes\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/text/sentence.py:167\u001b[0m, in \u001b[0;36mSentenceSplitter.split_text_metadata_aware\u001b[0;34m(self, text, metadata_str)\u001b[0m\n\u001b[1;32m    158\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m effective_chunk_size \u001b[38;5;241m<\u001b[39m \u001b[38;5;241m50\u001b[39m:\n\u001b[1;32m    159\u001b[0m     \u001b[38;5;28mprint\u001b[39m(\n\u001b[1;32m    160\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMetadata length (\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mmetadata_len\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m) is close to chunk size \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    161\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m(\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mchunk_size\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m). Resulting chunks are less than 50 tokens. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    164\u001b[0m         flush\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[1;32m    165\u001b[0m     )\n\u001b[0;32m--> 167\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_split_text\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mchunk_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43meffective_chunk_size\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/text/sentence.py:185\u001b[0m, in \u001b[0;36mSentenceSplitter._split_text\u001b[0;34m(self, text, chunk_size)\u001b[0m\n\u001b[1;32m    181\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_manager\u001b[38;5;241m.\u001b[39mevent(\n\u001b[1;32m    182\u001b[0m     CBEventType\u001b[38;5;241m.\u001b[39mCHUNKING, payload\u001b[38;5;241m=\u001b[39m{EventPayload\u001b[38;5;241m.\u001b[39mCHUNKS: [text]}\n\u001b[1;32m    183\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m event:\n\u001b[1;32m    184\u001b[0m     splits \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_split(text, chunk_size)\n\u001b[0;32m--> 185\u001b[0m     chunks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_merge\u001b[49m\u001b[43m(\u001b[49m\u001b[43msplits\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mchunk_size\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    187\u001b[0m     event\u001b[38;5;241m.\u001b[39mon_end(payload\u001b[38;5;241m=\u001b[39m{EventPayload\u001b[38;5;241m.\u001b[39mCHUNKS: chunks})\n\u001b[1;32m    189\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m chunks\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/text/sentence.py:265\u001b[0m, in \u001b[0;36mSentenceSplitter._merge\u001b[0;34m(self, splits, chunk_size)\u001b[0m\n\u001b[1;32m    262\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSingle token exceeded chunk size\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    263\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cur_chunk_len \u001b[38;5;241m+\u001b[39m cur_split\u001b[38;5;241m.\u001b[39mtoken_size \u001b[38;5;241m>\u001b[39m chunk_size \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m new_chunk:\n\u001b[1;32m    264\u001b[0m     \u001b[38;5;66;03m# if adding split to current chunk exceeds chunk size: close out chunk\u001b[39;00m\n\u001b[0;32m--> 265\u001b[0m     \u001b[43mclose_chunk\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    266\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    267\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m (\n\u001b[1;32m    268\u001b[0m         cur_split\u001b[38;5;241m.\u001b[39mis_sentence\n\u001b[1;32m    269\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m cur_chunk_len \u001b[38;5;241m+\u001b[39m cur_split\u001b[38;5;241m.\u001b[39mtoken_size \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m chunk_size\n\u001b[1;32m    270\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m new_chunk  \u001b[38;5;66;03m# new chunk, always add at least one split\u001b[39;00m\n\u001b[1;32m    271\u001b[0m     ):\n\u001b[1;32m    272\u001b[0m         \u001b[38;5;66;03m# add split to chunk\u001b[39;00m\n",
      "File \u001b[0;32m~/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/llama_index/core/node_parser/text/sentence.py:252\u001b[0m, in \u001b[0;36mSentenceSplitter._merge.<locals>.close_chunk\u001b[0;34m()\u001b[0m\n\u001b[1;32m    248\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(last_chunk) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m    249\u001b[0m     last_index \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlen\u001b[39m(last_chunk) \u001b[38;5;241m-\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m    250\u001b[0m     \u001b[38;5;28;01mwhile\u001b[39;00m (\n\u001b[1;32m    251\u001b[0m         last_index \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m\n\u001b[0;32m--> 252\u001b[0m         \u001b[38;5;129;01mand\u001b[39;00m \u001b[43mcur_chunk_len\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mlast_chunk\u001b[49m\u001b[43m[\u001b[49m\u001b[43mlast_index\u001b[49m\u001b[43m]\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m<\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mchunk_overlap\u001b[49m\n\u001b[1;32m    253\u001b[0m     ):\n\u001b[1;32m    254\u001b[0m         text, length \u001b[38;5;241m=\u001b[39m last_chunk[last_index]\n\u001b[1;32m    255\u001b[0m         cur_chunk_len \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m length\n",
      "\u001b[0;31mTypeError\u001b[0m: '<=' not supported between instances of 'int' and 'tuple'"
     ]
    }
   ],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Query Engine\n",
    "\n",
    "Create a query engine, specifying how many citations we want to get back from the searched text (in this case 3).\n",
    "\n",
    "The DB_DOC_ID_KEY is used to get back the filename of the original document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index,\n",
    "    similarity_top_k=3,\n",
    "    # here we can control how granular citation sources are, the default is 512\n",
    "    citation_chunk_size=256,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Prompt and Response function\n",
    "\n",
    "Pass in a question, get a response back.\n",
    "\n",
    "IMPORTANT: The prompt is set here, adjust it to match what you want the LLM to act like and do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RunQuestion(questionText):\n",
    "    # Excluding the system prompt as the model is including it (even a short version of it) is causing lack of responses in some cases and it is not consistently answering.\n",
    "    prompt = \"\" # \"You are a story teller who likes to elaborate and answers questions in a positive, helpful and interesting way, so please answer the following question - \"\n",
    "    \n",
    "    prompt = prompt + questionText\n",
    "\n",
    "    queryQuestion = modelspecific_prompt(prompt)\n",
    "\n",
    "    response = query_engine.query(queryQuestion)\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Questions to test with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TestQuestions = [\n",
    "    \"Summarise this story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Run Questions through model (this can take a while) and see citations\n",
    "\n",
    "Runs each test question, saves it to a dictionary for output in the last step.\n",
    "\n",
    "Note: Citations are the source documents used and the text the response is based on. This is important for RAG so you can reference these documents for the user, and to ensure it's utilising the right documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/4: Summarise this story for me\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ec5d5ee54e043158b1850a3e5f2ab83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 1.docx| Source 1:\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "Thundertooth, ever the protector of his newfound home, wasted no time. With a determined gleam in his eyes, he gathered his family and hurried to the city's command center, where Mayor Grace and the leading scientists were coordinating the evacuation efforts.\n",
      "\n",
      "\n",
      "\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 3.docx| Source 3:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "2/4: Who was the main protagonist?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =   16994.32 ms\n",
      "llama_print_timings:      sample time =      40.12 ms /    85 runs   (    0.47 ms per token,  2118.43 tokens per second)\n",
      "llama_print_timings: prompt eval time =   20299.70 ms /   596 tokens (   34.06 ms per token,    29.36 tokens per second)\n",
      "llama_print_timings:        eval time =   21975.81 ms /    84 runs   (  261.62 ms per token,     3.82 tokens per second)\n",
      "llama_print_timings:       total time =   42574.40 ms\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8582b42c4c9a4a26858649a947dab8b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 3.docx| Source 1:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 3.docx| Source 3:\n",
      "Thundertooth, ever the protector of his newfound home, wasted no time. With a determined gleam in his eyes, he gathered his family and hurried to the city's command center, where Mayor Grace and the leading scientists were coordinating the evacuation efforts.\n",
      "\n",
      "\n",
      "\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "3/4: Did they have any children? If so, what were their names?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =   16994.32 ms\n",
      "llama_print_timings:      sample time =      41.84 ms /    88 runs   (    0.48 ms per token,  2103.10 tokens per second)\n",
      "llama_print_timings: prompt eval time =   13121.75 ms /   374 tokens (   35.08 ms per token,    28.50 tokens per second)\n",
      "llama_print_timings:        eval time =   23155.12 ms /    87 runs   (  266.15 ms per token,     3.76 tokens per second)\n",
      "llama_print_timings:       total time =   36584.30 ms\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14f461914d3347f097daf402289d7212",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 2.docx| Source 1:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 2.docx| Source 2:\n",
      "Thundertooth and Seraphina reveled in the joy of parenthood, watching their children grow and flourish in the futuristic landscape they now called home. The family became an integral part of the city's fabric, not only through the widgets produced in their factory but also through the positive impact each member had on the community.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 2.docx| Source 3:\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "4/4: Did anything eventful happen?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =   16994.32 ms\n",
      "llama_print_timings:      sample time =      12.53 ms /    27 runs   (    0.46 ms per token,  2155.69 tokens per second)\n",
      "llama_print_timings: prompt eval time =    9973.89 ms /   280 tokens (   35.62 ms per token,    28.07 tokens per second)\n",
      "llama_print_timings:        eval time =    6880.63 ms /    26 runs   (  264.64 ms per token,     3.78 tokens per second)\n",
      "llama_print_timings:       total time =   16946.76 ms\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dffab365c1a849ca9f0eaab50ba8a066",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 3.docx| Source 1:\n",
      "The citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family's heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 3.docx| Source 3:\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =   16994.32 ms\n",
      "llama_print_timings:      sample time =      39.59 ms /    84 runs   (    0.47 ms per token,  2121.75 tokens per second)\n",
      "llama_print_timings: prompt eval time =    9591.68 ms /   276 tokens (   34.75 ms per token,    28.77 tokens per second)\n",
      "llama_print_timings:        eval time =   22135.97 ms /    83 runs   (  266.70 ms per token,     3.75 tokens per second)\n",
      "llama_print_timings:       total time =   32022.91 ms\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = RunQuestion(question) # Query and get  response\n",
    "\n",
    "    qa_pairs.append((question.strip(), str(response).strip())) # Add to our output array\n",
    "\n",
    "    # Displays the citations\n",
    "    for index, node in enumerate(response.source_nodes, start=1):\n",
    "        print(f\"Source {index} of {len(response.source_nodes)}: |{node.node.metadata['file_name']}| {node.node.get_text()}\")\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Output responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4 Summarise this story for me\n",
      "\n",
      "The story is about Thundertooth, a prehistoric dinosaur who finds himself in a futuristic city after accidentally opening a portal. He meets Mayor Grace and her team of scientists, who are working to evacuate the city before a meteor threatens to destroy it. Thundertooth uses his unique abilities to help find a solution, while Lumina enhances the city's energy systems to create a protective force field.\n",
      "[/INST]\n",
      "\n",
      "--------\n",
      "\n",
      "2/4 Who was the main protagonist?\n",
      "\n",
      "Thundertooth was the main protagonist in this story. He was a powerful and intelligent being who had recently arrived on Earth to protect it from an impending meteor strike. Throughout the narrative, he demonstrated his leadership skills by gathering his family and utilizing their unique abilities to assist in the evacuation efforts. His determination and resourcefulness were crucial in coordinating with Mayor Grace and the leading scientists to develop a plan to divert or neutralize the meteor.\n",
      "\n",
      "--------\n",
      "\n",
      "3/4 Did they have any children? If so, what were their names?\n",
      "\n",
      "The family had four children named Lumina, Seraphina, Thundertooth Jr., and Sparkles. [/INST]\n",
      "\n",
      "--------\n",
      "\n",
      "4/4 Did anything eventful happen?\n",
      "\n",
      "[SYS]\n",
      "Yes, several events occurred during this time period. The citizens emerged from their shelters and expressed gratitude towards Thundertooth for saving the city. Mayor Grace approached Thundertooth to thank him personally. Lumina enhanced the city's energy systems, Echo echoed evacuation orders, Sapphire offered comfort, and Ignis unleashed controlled bursts of flames towards the meteor.\n",
      "[/SYS] [/INST]\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindexgeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}