examples/system-demo/experiment_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Querying different models from the same endpoint: prompto vs. synchronous Python for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import os\n",
    "import tqdm\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "from prompto.settings import Settings\n",
    "from prompto.experiment import Experiment\n",
    "\n",
    "from api_utils import send_prompt\n",
    "from dataset_utils import (\n",
    "    load_prompt_dicts,\n",
    "    load_prompts,\n",
    "    generate_experiment_1_file,\n",
    "    generate_experiment_3_file,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this experiment, we want to compare the performance of `prompto` which uses asynchronous programming to query model API endpoints with a traditional synchronous Python for loop. For this experiment, we are going to compare the time it takes for `prompto` to obtain 100 responses from 3 different models over the same API endpoint and the time it takes for a synchronous Python for loop to obtain the same 100 responses for each model.\n",
    "\n",
    "We will see that `prompto` is able to obtain the responses from the models much faster than the synchronous Python for loop, especially when using parallel processing to query the models in parallel.\n",
    "\n",
    "We choose to query three different models from the Open API endpoint for this experiment: `gpt-3.5-turbo`, `gpt-4` and `gpt-4o`.\n",
    "\n",
    "For this experiment, we will need to set up the following environment variables:\n",
    "- `OPENAI_API_KEY`: the API key for the OpenAI API\n",
    "\n",
    "To set these environment variables, one can simply have these in a `.env` file which specifies these environment variables as key-value pairs:\n",
    "```\n",
    "OPENAI_API_KEY=<YOUR-OPENAI=KEY>\n",
    "```\n",
    "\n",
    "If you make this file, you can run the following which should return True if it's found one, or False otherwise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_dotenv(dotenv_path=\".env\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the experiment, we take a sample of 100 prompts from the [`alpaca_data.json`](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) from the [`tatsu-lab/stanford_alpaca` Github repo](https://github.com/tatsu-lab/stanford_alpaca) and using the prompt template provided by the authors of the repo. To see how we obtain the prompts, please refer to the [alpaca_sample_generation.ipynb](https://alan-turing-institute.github.io/prompto/examples/system-demo/alpaca_sample_generation/) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpaca_prompts = load_prompts(\"./sample_prompts.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synchronous approach\n",
    "\n",
    "For the synchronous approach, we simply use a for loop to query the API endpoints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def send_prompts_sync(prompt_dicts: list[dict]) -> list[str]:\n",
    "    # naive for loop to synchronously dispatch prompts\n",
    "    return [send_prompt(prompt_dict) for prompt_dict in tqdm(prompt_dicts)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying of models in parallel with `prompto`\n",
    "\n",
    "We first look at querying each of the models in parallel using `prompto`. \n",
    "\n",
    "### Experiment setup\n",
    "\n",
    "We will create our experiment files using the `generate_experiment_3_file` function in the `dataset_utils.py` file in this directory. This function will just take these prompts and create a jsonl file with the prompts in the format that `prompto` expects. We will save these input files into `./data/input` and use `./data` are our pipeline data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_MULTIPLE_EXPERIMENT_FILE = \"./data/input/openai-multiple-models.jsonl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_experiment_3_file(\n",
    "    path=OPENAI_MULTIPLE_EXPERIMENT_FILE,\n",
    "    prompts=alpaca_prompts,\n",
    "    api=\"openai\",\n",
    "    model_name=[\"gpt-3.5-turbo\", \"gpt-4\", \"gpt-4o\"],\n",
    "    params={\"n\": 1, \"temperature\": 0.9, \"max_tokens\": 100},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len(load_prompt_dicts(OPENAI_MULTIPLE_EXPERIMENT_FILE)): 300\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"len(load_prompt_dicts(OPENAI_MULTIPLE_EXPERIMENT_FILE)): \"\n",
    "    f\"{len(load_prompt_dicts(OPENAI_MULTIPLE_EXPERIMENT_FILE))}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 300/300 [11:45<00:00,  2.35s/it]\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "overall_sync = send_prompts_sync(\n",
    "    prompt_dicts=load_prompt_dicts(OPENAI_MULTIPLE_EXPERIMENT_FILE)\n",
    ")\n",
    "sync_times[\"overall\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying the rate limits for each model for parallel processing\n",
    "\n",
    "Notice here that we are setting `parallel=True` in the `Settings` object as well as specifying the rate limits to send to each of the models, and we specify each of them to be 500.  We do this by passing in a dictionary to the `max_queries_dict` argument in the `Settings` object which has API names as the keys and the values are also a dictionary where the keys are the model names we wish to process in parallel and the values are rate limits.\n",
    "\n",
    "For details of how to specify rate limits for _different models in the same API_, see the [Specifying rate limits docs](https://alan-turing-institute.github.io/prompto/docs/rate_limits/) and the [Grouping prompts and specifying rate limits notebook](https://alan-turing-institute.github.io/prompto/examples/notebooks/grouping_prompts_and_specifying_rate_limits/).\n",
    "\n",
    "Note that in the [previous experiment](https://alan-turing-institute.github.io/prompto/examples/system-demo/experiment_2/), we also used parallel processing but in a slightly different way as we were parallelising the querying of different APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Waiting for all groups to complete:   0%|          | 0/4 [00:00<?, ?group/s]\n",
      "Sending 0 queries at 10 QPM with RI of 6.0s for group openai  (attempt 1/3): 0query [00:00, ?query/s]\n",
      "Waiting for responses for group openai  (attempt 1/3): 0query [00:00, ?query/s]\n",
      "Sending 100 queries at 500 QPM with RI of 0.12s for group openai-gpt-3.5-turbo  (attempt 1/3): 100%|██████████| 100/100 [00:12<00:00,  7.73query/s]\n",
      "Sending 100 queries at 500 QPM with RI of 0.12s for group openai-gpt-4  (attempt 1/3): 100%|██████████| 100/100 [00:12<00:00,  7.72query/s]\n",
      "Sending 100 queries at 500 QPM with RI of 0.12s for group openai-gpt-4o  (attempt 1/3): 100%|██████████| 100/100 [00:13<00:00,  7.69query/s]\n",
      "Waiting for responses for group openai-gpt-3.5-turbo  (attempt 1/3): 100%|██████████| 100/100 [00:01<00:00, 57.16query/s]\n",
      "Waiting for all groups to complete:  50%|█████     | 2/4 [00:14<00:14,  7.35s/group]\n",
      "Waiting for responses for group openai-gpt-4o  (attempt 1/3): 100%|██████████| 100/100 [00:03<00:00, 26.66query/s]\n",
      "Waiting for all groups to complete:  75%|███████▌  | 3/4 [00:16<00:05,  5.15s/group]\n",
      "Waiting for responses for group openai-gpt-4  (attempt 1/3): 100%|██████████| 100/100 [00:06<00:00, 15.87query/s]\n",
      "Waiting for all groups to complete: 100%|██████████| 4/4 [00:19<00:00,  4.82s/group]\n"
     ]
    }
   ],
   "source": [
    "gpt4o_experiment = Experiment(\n",
    "    file_name=\"openai-multiple-models.jsonl\",\n",
    "    settings=Settings(\n",
    "        data_folder=\"./data\",\n",
    "        parallel=True,\n",
    "        max_queries_dict={\n",
    "            \"openai\": {\"gpt-3.5-turbo\": 500, \"gpt-4\": 500, \"gpt-4o\": 500}\n",
    "        },\n",
    "    ),\n",
    ")\n",
    "\n",
    "start = time.time()\n",
    "gpt4o_responses, _ = await gpt4o_experiment.process()\n",
    "prompto_times[\"overall\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running `prompto` via the command line\n",
    "\n",
    "We could have also ran this experiment with `prompto` via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. `examples/system-demo`):\n",
    "```bash\n",
    "prompto_run_experiment --file data/input/openai-multiple-models.jsonl --parallel True --max-queries-json experiment_3_parallel_config.json\n",
    "```\n",
    "where `experiment_3_parallel_config.json` is a JSON file that specifies the rate limits for each of the API endpoints:\n",
    "```json\n",
    "{\n",
    "    \"openai\": {\n",
    "        \"gpt-3.5-turbo\": 500,\n",
    "        \"gpt-4\": 500,\n",
    "        \"gpt-4o\": 500\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "But for this notebook, we will time the experiments and save them to the `sync_times` and `prompto_times` dictionaries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying the models without parallel processing\n",
    "\n",
    "We will also compare the runtime to obtain responses from each of the models using a synchronous Python for loop versus using `prompto` to query the models asynchronously _without_ parallel processing. We will look at using parallel processing in a [later section](#querying-of-models-in-parallel-with-prompto).\n",
    "\n",
    "### Experiment setup\n",
    "\n",
    "We will create our experiment files using the `generate_experiment_1_file` function in the `dataset_utils.py` file in this directory. This function will just take these prompts and create a jsonl file with the prompts in the format that `prompto` expects. We will save these input files into `./data/input` and use `./data` are our pipeline data folder.\n",
    "\n",
    "See the [pipeline data docs](https://alan-turing-institute.github.io/prompto/docs/pipeline/) for more information about the pipeline data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_GPT_35_TURBO_EXPERIMENT_FILE = \"./data/input/openai-gpt-3pt5-turbo.jsonl\"\n",
    "OPENAI_GPT_4_EXPERIMENT_FILE = \"./data/input/openai-gpt-4.jsonl\"\n",
    "OPENAI_GPT_4O_EXPERIMENT_FILE = \"./data/input/openai-gpt-4o.jsonl\"\n",
    "\n",
    "INPUT_EXPERIMENT_FILEDIR = \"./data/input\"\n",
    "\n",
    "if not os.path.isdir(INPUT_EXPERIMENT_FILEDIR):\n",
    "    os.mkdir(INPUT_EXPERIMENT_FILEDIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_experiment_1_file(\n",
    "    path=OPENAI_GPT_35_TURBO_EXPERIMENT_FILE,\n",
    "    prompts=alpaca_prompts,\n",
    "    api=\"openai\",\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    "    params={\"n\": 1, \"temperature\": 0.9, \"max_tokens\": 100},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_experiment_1_file(\n",
    "    path=OPENAI_GPT_4_EXPERIMENT_FILE,\n",
    "    prompts=alpaca_prompts,\n",
    "    api=\"openai\",\n",
    "    model_name=\"gpt-4\",\n",
    "    params={\"n\": 1, \"temperature\": 0.9, \"max_tokens\": 100},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_experiment_1_file(\n",
    "    path=OPENAI_GPT_4O_EXPERIMENT_FILE,\n",
    "    prompts=alpaca_prompts,\n",
    "    api=\"openai\",\n",
    "    model_name=\"gpt-4o\",\n",
    "    params={\"n\": 1, \"temperature\": 0.9, \"max_tokens\": 100},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each model, we will compare runtimes for using `prompto` and a synchronous Python for loop to obtain 100 responses from the model.\n",
    "\n",
    "We use the `send_prompts_sync` function defined above for the synchronous Python for loop approach. We can run experiments using the `prompto.experiment.Experiment.process` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sync_times = {}\n",
    "prompto_times = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len(load_prompt_dicts(OPENAI_GPT_35_TURBO_EXPERIMENT_FILE)): 100\n",
      "len(load_prompt_dicts(OPENAI_GPT_4_EXPERIMENT_FILE)): 100\n",
      "len(load_prompt_dicts(OPENAI_GPT_4O_EXPERIMENT_FILE)): 100\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"len(load_prompt_dicts(OPENAI_GPT_35_TURBO_EXPERIMENT_FILE)): {len(load_prompt_dicts(OPENAI_GPT_35_TURBO_EXPERIMENT_FILE))}\"\n",
    ")\n",
    "print(\n",
    "    f\"len(load_prompt_dicts(OPENAI_GPT_4_EXPERIMENT_FILE)): {len(load_prompt_dicts(OPENAI_GPT_4_EXPERIMENT_FILE))}\"\n",
    ")\n",
    "print(\n",
    "    f\"len(load_prompt_dicts(OPENAI_GPT_4O_EXPERIMENT_FILE)): {len(load_prompt_dicts(OPENAI_GPT_4O_EXPERIMENT_FILE))}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPT-3.5-turbo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment synchronously"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [02:10<00:00,  1.31s/it]\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "openai_sync = send_prompts_sync(\n",
    "    prompt_dicts=load_prompt_dicts(OPENAI_GPT_35_TURBO_EXPERIMENT_FILE)\n",
    ")\n",
    "sync_times[\"gpt-3.5-turbo\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment asynchronously with `prompto`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending 100 queries at 500 QPM with RI of 0.12s  (attempt 1/3):   0%|          | 0/100 [00:00<?, ?query/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending 100 queries at 500 QPM with RI of 0.12s  (attempt 1/3): 100%|██████████| 100/100 [00:12<00:00,  8.20query/s]\n",
      "Waiting for responses  (attempt 1/3): 100%|██████████| 100/100 [00:02<00:00, 48.86query/s]\n"
     ]
    }
   ],
   "source": [
    "openai_experiment = Experiment(\n",
    "    file_name=\"openai-gpt-3pt5-turbo.jsonl\",\n",
    "    settings=Settings(data_folder=\"./data\", max_queries=500),\n",
    ")\n",
    "\n",
    "start = time.time()\n",
    "openai_responses, _ = await openai_experiment.process()\n",
    "prompto_times[\"gpt-3.5-turbo\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(130.72972202301025, 14.28760814666748)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sync_times[\"gpt-3.5-turbo\"], prompto_times[\"gpt-3.5-turbo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPT-4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment synchronously"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [06:32<00:00,  3.92s/it]\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "gpt4_sync = send_prompts_sync(\n",
    "    prompt_dicts=load_prompt_dicts(OPENAI_GPT_4_EXPERIMENT_FILE)\n",
    ")\n",
    "sync_times[\"gpt4\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment asynchronously with `prompto`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending 100 queries at 500 QPM with RI of 0.12s  (attempt 1/3): 100%|██████████| 100/100 [00:12<00:00,  8.14query/s]\n",
      "Waiting for responses  (attempt 1/3): 100%|██████████| 100/100 [00:07<00:00, 13.36query/s]\n"
     ]
    }
   ],
   "source": [
    "gpt4_experiment = Experiment(\n",
    "    file_name=\"openai-gpt-4.jsonl\",\n",
    "    settings=Settings(data_folder=\"./data\", max_queries=500),\n",
    ")\n",
    "\n",
    "start = time.time()\n",
    "gpt4_responses, _ = await gpt4_experiment.process()\n",
    "prompto_times[\"gpt4\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(392.211834192276, 19.79161500930786)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sync_times[\"gpt4\"], prompto_times[\"gpt4\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPT-4o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment synchronously"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [04:01<00:00,  2.41s/it]\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "gpt4o_sync = send_prompts_sync(\n",
    "    prompt_dicts=load_prompt_dicts(OPENAI_GPT_4O_EXPERIMENT_FILE)\n",
    ")\n",
    "sync_times[\"gpt4o\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment asynchronously with `prompto`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending 100 queries at 500 QPM with RI of 0.12s  (attempt 1/3): 100%|██████████| 100/100 [00:12<00:00,  8.14query/s]\n",
      "Waiting for responses  (attempt 1/3): 100%|██████████| 100/100 [00:05<00:00, 17.25query/s]\n"
     ]
    }
   ],
   "source": [
    "gpt4o_experiment = Experiment(\n",
    "    file_name=\"openai-gpt-4o.jsonl\",\n",
    "    settings=Settings(data_folder=\"./data\", max_queries=500),\n",
    ")\n",
    "\n",
    "start = time.time()\n",
    "gpt4o_responses, _ = await gpt4o_experiment.process()\n",
    "prompto_times[\"gpt4o\"] = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(241.2371437549591, 18.114527940750122)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sync_times[\"gpt4o\"], prompto_times[\"gpt4o\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running `prompto` via the command line\n",
    "\n",
    "We can also run the above experiments via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. `examples/system-demo`):\n",
    "```bash\n",
    "prompto_run_experiment --file data/input/openai-gpt-3pt5-turbo.jsonl --max-queries 500\n",
    "prompto_run_experiment --file data/input/openai-gpt-4.jsonl --max-queries 500\n",
    "prompto_run_experiment --file data/input/openai-gpt-4o.jsonl --max-queries 500\n",
    "```\n",
    "\n",
    "But for this notebook, we will time the experiments and save them to the `sync_times` and `prompto_times` dictionaries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "Here, we report the final runtimes for each model and the difference in time between the `prompto` and synchronous Python for loop approaches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'gpt-3.5-turbo': 130.72972202301025,\n",
       " 'gpt4': 392.211834192276,\n",
       " 'gpt4o': 241.2371437549591,\n",
       " 'overall': 705.384626865387}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sync_times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'gpt-3.5-turbo': 14.28760814666748,\n",
       " 'gpt4': 19.79161500930786,\n",
       " 'gpt4o': 18.114527940750122,\n",
       " 'overall': 19.298332929611206}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompto_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see here a significant speed up within model (i.e. direct comparison of using `prompto` vs. synchronous Python for loop for a specific model) as well as across models (i.e. comparison of using `prompto` with parallel processing vs. synchronous Python for loop for different models). We see the runtime for parallel processing is just the time it takes to query the model with the longest runtime (in this case GPT-4)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}