pyttitools-PYTTI_local.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WksDx4_To_su"
      },
      "source": [
        "# PYTTI: Python Text-to-Image \n",
        "\n",
        "AI-Assisted Artistic Image Generation and Manipulation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZt160ePEc8f"
      },
      "source": [
        "# 1. Setup\n",
        "\n",
        "Setup instructions can be found here: https://pytti-tools.github.io/pytti-book/intro.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PcayyBJjE-qy"
      },
      "source": [
        "# 2: Define Parameters (i.e. the inputs to this tool)\n",
        "\n",
        "Settings documentation can be found here:\n",
        "* [Scene Syntax](https://pytti-tools.github.io/pytti-book/SceneDSL.html)\n",
        "* [Everything Else](https://pytti-tools.github.io/pytti-book/Settings.html)\n",
        "\n",
        "Please modify the values in the `config/` folder's yaml files, the files will be read when you execute the main \"workhorse\" cell.\n",
        "\n",
        "Feel free to modify the values in `config/default.yaml`. If you have discovered alternative default settings that you prefer, feel free to persist them in that file. You can then specify your runs by authoring files that define just the parameters you want to change in the `config/conf/` folder. An example `demo.yaml` is provided. \n",
        "\n",
        "In the following cell, we specify the name of the config file containing the overrides to the default settings. To use override settings other than those contained in `config/conf/demo.yaml`, specify the correct filename below. \n",
        "\n",
        "Descriptions of parameter options and features can be found at the bottom of this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# If multiple filenames are present, they will be looped\n",
        "CONFIG_OVERRIDES = [] # [\"demo.yaml\"] # File(s) located in config/conf\n",
        "\n",
        "# You probably don't need to touch these\n",
        "CONFIG_BASE_PATH = \"config\"\n",
        "CONFIG_DEFAULTS = \"default.yaml\"\n",
        "\n",
        "# If randomize_seed is set to False, two runs with the same parameters will produce the same outputs\n",
        "#RANDOMIZE_SEED = False # feature not yet supported.\n",
        "\n",
        "##################\n",
        "\n",
        "## Don't change values in this cell below this line\n",
        "\n",
        "# overrides to let you paste raw json\n",
        "null=None\n",
        "true=True\n",
        "false=False\n",
        "\n",
        "# added to allow users not pasting in settings to skip that cell\n",
        "pytti_panna_output = {}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you would like to use parameters generated by pyttipanna or an older version of the notebook, incorporate them below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Replace '{}' with your pytti panna thing\n",
        "pytti_panna_output = {}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 3. (Optional) Monitor Outputs with Tensorboard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook supports tensorboard outputs. These are the same values that will be printed while the notebook runs, but they may be easier to navigate in tensorboard if you want to start a server. Uncomment the cell below to initialize tensorboard in the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#from pytti.workhorse import _main, TB_LOGDIR \n",
        "\n",
        "#%load_ext tensorboard\n",
        "#%tensorboard --logdir $TB_LOGDIR "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 4. RUN IT!\n",
        "\n",
        "Just execute the cell below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from hydra import initialize, initialize_config_module, initialize_config_dir, compose\n",
        "from loguru import logger\n",
        "from omegaconf import OmegaConf\n",
        "from pytti.workhorse import _main as render_frames\n",
        "\n",
        "# Necessary for VS Code to respect displayed outputs\n",
        "%matplotlib inline\n",
        "\n",
        "try:\n",
        "    assert isinstance(CONFIG_OVERRIDES, list)\n",
        "except AssertionError:\n",
        "    if isinstance(CONFIG_OVERRIDES, str):\n",
        "        logger.warning(\n",
        "            \"The CONFIG_OVERRIDES variable should be a list of filenames.\"\n",
        "            \"I noticed you set this variable to a string. I'll wrap that in \"\n",
        "            \"a list for you this time, but after this cell completes execution, \"\n",
        "            \"please repair how you set the variable above. Instead of\"\n",
        "            f'\\n\\n\\tCONFIG_OVERRIDES=\"{CONFIG_OVERRIDES}\\n\\n'\n",
        "            \"it should be\"\n",
        "            f'\\n\\n\\tCONFIG_OVERRIDES=[\"{CONFIG_OVERRIDES}\"]\\n\\n'\n",
        "        )\n",
        "        CONFIG_OVERRIDES = [CONFIG_OVERRIDES]\n",
        "\n",
        "# https://github.com/facebookresearch/hydra/blob/main/examples/jupyter_notebooks/compose_configs_in_notebook.ipynb\n",
        "# https://omegaconf.readthedocs.io/\n",
        "# https://hydra.cc/docs/intro/\n",
        "with initialize(config_path=CONFIG_BASE_PATH):\n",
        "    if pytti_panna_output:\n",
        "        cfg = OmegaConf.create(pytti_panna_output)\n",
        "        render_frames(cfg)\n",
        "    for c_o in CONFIG_OVERRIDES:\n",
        "        cfg = compose(config_name=CONFIG_DEFAULTS, \n",
        "                      overrides=[f\"conf={c_o}\"])\n",
        "        render_frames(cfg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AtekvTZxFNpf"
      },
      "source": [
        "# 4. Render video\n",
        "\n",
        "The first run executed in `file_namespace` (probably set to `default`) is number $0$, the second is number $1$, etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "RZH-r4yyShnX"
      },
      "outputs": [],
      "source": [
        "# quick fix\n",
        "import re\n",
        "params = cfg\n",
        "\n",
        "from os.path import exists as path_exists\n",
        "#if path_exists('/content/drive/MyDrive/pytti_test'):\n",
        "#  %cd /content/drive/MyDrive/pytti_test\n",
        "#  drive_mounted = True\n",
        "#else:\n",
        "drive_mounted = False\n",
        "#try:\n",
        "#  from pytti.Notebook import change_tqdm_color\n",
        "#except ModuleNotFoundError:\n",
        "#  if drive_mounted:\n",
        "#    #THIS IS NOT AN ERROR. This is the code that would\n",
        "#    #make an error if something were wrong.\n",
        "#    raise RuntimeError('ERROR: please run setup (step 1.3).')\n",
        "#  else:\n",
        "#    #THIS IS NOT AN ERROR. This is the code that would\n",
        "#    #make an error if something were wrong.\n",
        "#    raise RuntimeError('WARNING: drive is not mounted.\\nERROR: please run setup (step 1.3).')\n",
        "#change_tqdm_color()\n",
        "  \n",
        "from tqdm.notebook import tqdm\n",
        "import numpy as np\n",
        "from os.path import exists as path_exists\n",
        "from subprocess import Popen, PIPE\n",
        "from PIL import Image, ImageFile\n",
        "from os.path import splitext as split_file\n",
        "import glob\n",
        "from pytti.Notebook import get_last_file\n",
        "\n",
        "ImageFile.LOAD_TRUNCATED_IMAGES = True\n",
        "\n",
        "try:\n",
        "  params\n",
        "except NameError:\n",
        "  raise RuntimeError(\"ERROR: no parameters. Please run parameters (step 2.1).\")\n",
        "\n",
        "if not path_exists(f\"images_out/{params.file_namespace}\"):\n",
        "  if path_exists(f\"/content/drive/MyDrive\"):\n",
        "    #THIS IS NOT AN ERROR. This is the code that would\n",
        "    #make an error if something were wrong.\n",
        "    raise RuntimeError(f\"ERROR: file_namespace: {params.file_namespace} does not exist.\")\n",
        "  else:\n",
        "    #THIS IS NOT AN ERROR. This is the code that would\n",
        "    #make an error if something were wrong.\n",
        "    raise RuntimeError(f\"WARNING: Drive is not mounted.\\nERROR: file_namespace: {params.file_namespace} does not exist.\")\n",
        "\n",
        "#@markdown The first run executed in `file_namespace` is number $0$, the second is number $1$, etc.\n",
        "\n",
        "latest = -1\n",
        "run_number = latest#@param{type:\"raw\"}\n",
        "if run_number == -1:\n",
        "  _, i = get_last_file(f'images_out/{params.file_namespace}', \n",
        "                       f'^(?P<pre>{re.escape(params.file_namespace)}\\\\(?)(?P<index>\\\\d*)(?P<post>\\\\)?_1\\\\.png)$')\n",
        "  run_number = i\n",
        "base_name = params.file_namespace if run_number == 0 else (params.file_namespace+f\"({run_number})\")\n",
        "tqdm.write(f'Generating video from {params.file_namespace}/{base_name}_*.png')\n",
        "\n",
        "all_frames = glob.glob(f'images_out/{params.file_namespace}/{base_name}_*.png')\n",
        "all_frames.sort(key = lambda s: int(split_file(s)[0].split('_')[-1]))\n",
        "print(f'found {len(all_frames)} frames matching images_out/{params.file_namespace}/{base_name}_*.png')\n",
        "\n",
        "start_frame = 0#@param{type:\"number\"}\n",
        "all_frames = all_frames[start_frame:]\n",
        "\n",
        "fps =  params.frames_per_second#@param{type:\"raw\"}\n",
        "\n",
        "total_frames = len(all_frames)\n",
        "\n",
        "if total_frames == 0:\n",
        "  #THIS IS NOT AN ERROR. This is the code that would\n",
        "  #make an error if something were wrong.\n",
        "  raise RuntimeError(f\"ERROR: no frames to render in images_out/{params.file_namespace}\")\n",
        "\n",
        "frames = []\n",
        "\n",
        "for filename in tqdm(all_frames):\n",
        "  frames.append(Image.open(filename))\n",
        "\n",
        "p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-r', str(fps), '-pix_fmt', 'yuv420p', '-crf', '1', '-preset', 'veryslow', f\"videos/{base_name}.mp4\"], stdin=PIPE)\n",
        "for im in tqdm(frames):\n",
        "  im.save(p.stdin, 'PNG')\n",
        "p.stdin.close()\n",
        "\n",
        "print(\"Encoding video...\")\n",
        "p.wait()\n",
        "print(\"Video complete.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "t3EgqHKrSjZx"
      },
      "outputs": [],
      "source": [
        "#@title 3.1 Render video (concatenate all runs)\n",
        "#from os.path import exists as path_exists\n",
        "#if path_exists('/content/drive/MyDrive/pytti_test'):\n",
        "#  %cd /content/drive/MyDrive/pytti_test\n",
        "#  drive_mounted = True\n",
        "#else:\n",
        "#  drive_mounted = False\n",
        "#try:\n",
        "#  from pytti.Notebook import change_tqdm_color\n",
        "#except ModuleNotFoundError:\n",
        "#  if drive_mounted:\n",
        "#    #THIS IS NOT AN ERROR. This is the code that would\n",
        "#    #make an error if something were wrong.\n",
        "#    raise RuntimeError('ERROR: please run setup (step 1.3).')\n",
        "#  else:\n",
        "#    #THIS IS NOT AN ERROR. This is the code that would\n",
        "#    #make an error if something were wrong.\n",
        "#    raise RuntimeError('WARNING: drive is not mounted.\\nERROR: please run setup (step 1.3).')\n",
        "#change_tqdm_color()\n",
        "  \n",
        "from tqdm.notebook import tqdm\n",
        "import numpy as np\n",
        "from os.path import exists as path_exists\n",
        "from subprocess import Popen, PIPE\n",
        "from PIL import Image, ImageFile\n",
        "from os.path import splitext as split_file\n",
        "import glob\n",
        "from pytti.Notebook import get_last_file\n",
        "\n",
        "ImageFile.LOAD_TRUNCATED_IMAGES = True\n",
        "\n",
        "try:\n",
        "  params\n",
        "except NameError:\n",
        "  raise RuntimeError(\"ERROR: no parameters. Please run parameters (step 2.1).\")\n",
        "\n",
        "#if not path_exists(f\"images_out/{params.file_namespace}\"):\n",
        "#  if path_exists(f\"/content/drive/MyDrive\"):\n",
        "#    raise RuntimeError(f\"ERROR: file_namespace: {params.file_namespace} does not exist.\")\n",
        "#  else:\n",
        "#    raise RuntimeError(f\"WARNING: Drive is not mounted.\\nERROR: file_namespace: {params.file_namespace} does not exist.\")\n",
        "\n",
        "#@markdown The first run executed in `file_namespace` is number $0$, the second is number $1$, etc.\n",
        "\n",
        "latest = -1\n",
        "run_number = latest\n",
        "if run_number == -1:\n",
        "  _, i = get_last_file(f'images_out/{params.file_namespace}', \n",
        "                       f'^(?P<pre>{re.escape(params.file_namespace)}\\\\(?)(?P<index>\\\\d*)(?P<post>\\\\)?_1\\\\.png)$')\n",
        "  run_number = i\n",
        "\n",
        "all_frames = []\n",
        "for i in range(run_number+1):\n",
        "  base_name = params.file_namespace if i == 0 else (params.file_namespace+f\"({i})\")\n",
        "  frames = glob.glob(f'images_out/{params.file_namespace}/{base_name}_*.png')\n",
        "  frames.sort(key = lambda s: int(split_file(s)[0].split('_')[-1]))\n",
        "  all_frames.extend(frames)\n",
        "\n",
        "start_frame = 0#@param{type:\"number\"}\n",
        "all_frames = all_frames[start_frame:]\n",
        "\n",
        "fps =  params.frames_per_second#@param{type:\"raw\"}\n",
        "\n",
        "total_frames = len(all_frames)\n",
        "\n",
        "if total_frames == 0:\n",
        "  #THIS IS NOT AN ERROR. This is the code that would\n",
        "  #make an error if something were wrong.\n",
        "  raise RuntimeError(f\"ERROR: no frames to render in images_out/{params.file_namespace}\")\n",
        "\n",
        "frames = []\n",
        "\n",
        "for filename in tqdm(all_frames):\n",
        "  frames.append(Image.open(filename))\n",
        "\n",
        "p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-r', str(fps), '-pix_fmt', 'yuv420p', '-crf', '1', '-preset', 'veryslow', f\"videos/{base_name}.mp4\"], stdin=PIPE)\n",
        "for im in tqdm(frames):\n",
        "  im.save(p.stdin, 'PNG')\n",
        "p.stdin.close()\n",
        "\n",
        "print(\"Encoding video...\")\n",
        "p.wait()\n",
        "print(\"Video complete.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "-qZ8c_-iZ0QM"
      },
      "outputs": [],
      "source": [
        "# Optionally display your video in the notebook\n",
        "\n",
        "from IPython.display import Video\n",
        "\n",
        "output_video_filename = \"default.mp4\" # you'll probably need to change this after the first run\n",
        "\n",
        "Video(output_video_filename)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "if_Fdy_OFkjZ"
      },
      "source": [
        "# Batch Setings\n",
        "\n",
        "Get fancy with a for loop or try using hydra's `--multi-run` feature. Improved documentation for lcoal batch jobs forthcoming."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Debugging tools"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Run this cell if you think your environment is broken\n",
        "\n",
        "%conda info --envs\n",
        "\n",
        "import warnings\n",
        "\n",
        "try:\n",
        "    import torch\n",
        "    torch.cuda.is_available()\n",
        "except ImportError as e:\n",
        "    warnings.warn(\"Please install pytorch with CUDA (GPU): https://pytorch.org/get-started/locally/\")\n",
        "    warnings.warn(\"This is probably what you want: \")\n",
        "\n",
        "try:\n",
        "    import PIL\n",
        "except ImportError as e:\n",
        "    warnings.warn(\"Please install the Python Image Library: conda install -c conda-forge pillow\")\n",
        "\n",
        "try:\n",
        "    import cv2\n",
        "except ImportError as e:\n",
        "    warnings.warn(\"Please install OpenCV: conda install -c conda-forge opencv\")\n",
        "\n",
        "try:\n",
        "    import tensorflow as tf\n",
        "except ImportError as e:\n",
        "    warnings.warn(\"Please install Tensorflow with CUDA (GPU): conda install tensorflow-gpu\")\n",
        "\n",
        "try:\n",
        "    import transformers\n",
        "except:\n",
        "    warnings.warn(\"Please install huggingface/transformers: conda install -c huggingface transformers\")\n",
        "\n",
        "\n",
        "#conda install -c conda-forge imageio\n",
        "#conda install scikit-learn\n",
        "#pip install gdown einops\n",
        "#conda install pytorch-lightning -c conda-forge\n",
        "#conda install -c conda-forge kornia\n",
        "#conda install pandas \n",
        "#conda install seaborn\n",
        "#pip install PyGLM\n",
        "#pip install ftfy regex tqdm hydra-core adjustText exrex bunch matplotlib-label-lines\n",
        "#git clone https://github.com/pytti-tools/pytti-core\n",
        "#pip install ./AdaBins\n",
        "#pip install ./GMA\n",
        "#pip install ./CLIP\n",
        "#pip install ./pytti-core"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "pytti 5 beta.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}