GPT_with_vision_for_video_understanding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing and narrating a video with GPT's visual capabilities and the TTS API\n",
    "\n",
    "This notebook demonstrates how to use GPT's visual capabilities with a video. GPT-4 doesn't take videos as input directly, but we can use vision and the new 128K context window to describe the static frames of a whole video at once. We'll walk through two examples:\n",
    "\n",
    "1. Using GPT-4 to get a description of a video\n",
    "2. Generating a voiceover for a video with GPT-4 and the TTS API\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Image, Audio\n",
    "\n",
    "import cv2  # We're using OpenCV to read video\n",
    "import base64\n",
    "import time\n",
    "import openai\n",
    "import os\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Using GPT's visual capabilities to get a description of a video\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we use OpenCV to extract frames from a nature [video](https://www.youtube.com/watch?v=kQ_7GtE529M) containing bisons and wolves:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video = cv2.VideoCapture(\"data/bison.mp4\")\n",
    "\n",
    "base64Frames = []\n",
    "while video.isOpened():\n",
    "    success, frame = video.read()\n",
    "    if not success:\n",
    "        break\n",
    "    _, buffer = cv2.imencode(\".jpg\", frame)\n",
    "    base64Frames.append(base64.b64encode(buffer).decode(\"utf-8\"))\n",
    "\n",
    "video.release()\n",
    "print(len(base64Frames), \"frames read.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display frames to make sure we've read them in correctly:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_handle = display(None, display_id=True)\n",
    "for img in base64Frames:\n",
    "    display_handle.update(Image(data=base64.b64decode(img.encode(\"utf-8\"))))\n",
    "    time.sleep(0.025)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have the video frames we craft our prompt and send a request to GPT (Note that we don't need to send every frame for GPT to understand what's going on):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_MESSAGES = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            \"These are frames from a video that I want to upload. Generate a compelling description that I can upload along with the video.\",\n",
    "            *map(lambda x: {\"image\": x, \"resize\": 768}, base64Frames[0::10]),\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "params = {\n",
    "    \"model\": \"gpt-4-vision-preview\",\n",
    "    \"messages\": PROMPT_MESSAGES,\n",
    "    \"api_key\": os.environ[\"OPENAI_API_KEY\"],\n",
    "    \"headers\": {\"Openai-Version\": \"2020-11-07\"},\n",
    "    \"max_tokens\": 200,\n",
    "}\n",
    "\n",
    "result = openai.ChatCompletion.create(**params)\n",
    "print(result.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Generating a voiceover for a video with GPT-4 and the TTS API\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a voiceover for this video in the style of David Attenborough. Using the same video frames we prompt GPT to give us a short script:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_MESSAGES = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            \"These are frames of a video. Create a short voiceover script in the style of David Attenborough. Only include the narration.\",\n",
    "            *map(lambda x: {\"image\": x, \"resize\": 768}, base64Frames[0::10]),\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "params = {\n",
    "    \"model\": \"gpt-4-vision-preview\",\n",
    "    \"messages\": PROMPT_MESSAGES,\n",
    "    \"api_key\": os.environ[\"OPENAI_API_KEY\"],\n",
    "    \"headers\": {\"Openai-Version\": \"2020-11-07\"},\n",
    "    \"max_tokens\": 500,\n",
    "}\n",
    "\n",
    "result = openai.ChatCompletion.create(**params)\n",
    "print(result.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can pass the script to the TTS API where it will generate a mp3 of the voiceover:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "    \"https://api.openai.com/v1/audio/speech\",\n",
    "    headers={\n",
    "        \"Authorization\": f\"Bearer {os.environ['OPENAI_API_KEY']}\",\n",
    "    },\n",
    "    json={\n",
    "        \"model\": \"tts-1\",\n",
    "        \"input\": result.choices[0].message.content,\n",
    "        \"voice\": \"onyx\",\n",
    "    },\n",
    ")\n",
    "\n",
    "audio = b\"\"\n",
    "for chunk in response.iter_content(chunk_size=1024 * 1024):\n",
    "    audio += chunk\n",
    "Audio(audio)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}