content/Chapter1/Part2-图像生成方法/2-4-2ControlNet[WebUI].ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0 章节目标\n",
    "- 了解ControlNet的基本原理\n",
    "- 了解ControlNet的基本参数设置\n",
    "- 了解ControlNet在SD Web UI中的实战"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 概念介绍\n",
    "从2022年上半年开始，AI绘画的出世吸引了业内很多人的眼球。Stable Diffusion大模型的开源，使得AI绘画的应用变得空前火热。虽然Stable Diffusion生成的图片质量远超以往的GAN，VAE等，但还达不到精细化可控生成的目的。ControlNet在Stable Diffusion的基础上加入了更多可控生成方式，在AI绘图大规模商用上看到了曙光，可以说再次带热了AI绘图设计。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 基本原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ControlNet 网络设计\n",
    "在一个扩散模型中，如果不加ControlNet的扩散模型，其中原始的神经网络$F$输入$x$获得$y$，参数用$\\Theta $表示。  \n",
    "$$y=F(x;\\Theta )$$  \n",
    "也就是下图所示：  \n",
    "\n",
    "![](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/a566254b81f9481e8a229c7287d873b9~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)  \n",
    "\n",
    "ControlNet中，就是将模型原始的神经网络锁定，设为locked copy。  \n",
    "然后将原始网络的模型复制一份，称之为trainable copy，在其上进行操作施加控制条件。然后将施加控制条件之后的结果和原来模型的结果相加获得最终的输出。  \n",
    "最后将原始网络的输出修改为：  \n",
    "$$ y_{c} = F(x;\\Theta )+Z(F(x+Z(c;\\Theta _{z1} );\\Theta _{c} );\\Theta _{z2} ) $$  \n",
    "其中zero convolution，也就是零卷积层$Z$是初始化weight和bias为0，两层零卷积的参数为$\\left \\{ \\Theta _{z1} ,\\Theta _{z2}  \\right \\} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b887ea05e29446d0aae80528b8299a27~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将控制条件通过零卷积之后，与原始输入相加，相加之后进入ControlNet的复制神经网络块中，将网络输出再做一次零卷积之后与原始网络的输出相加。  \n",
    "初始化之后未经训练的ControlNet参数应该是这样的："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\left\\{\\begin{matrix}\n",
    " Z(c;\\Theta _{z1} )=0\\\\F(x+Z(c;\\Theta _{z1} );\\Theta _{c} )=F(x;\\Theta _{c} )=F(x;\\Theta )\n",
    " \\\\Z(F(x+Z(c;\\Theta _{z1} );\\Theta _{c} );\\Theta _{z2} )=Z(F(x;\\Theta _{c} );\\Theta _{z2} )=0\n",
    "\\end{matrix}\\right.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "也就是说ControlNet未经训练的时候，输出为0，那加到原始网络上的数字也是0。这样对原始网络是没有任何影响的，就能确保原网络的性能得以完整保存。之后ControlNet训练也只是在原始网络上进行优化，这样可以认为和微调网络是一样的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ControlNet in Stable Diffusion\n",
    "上一部分描述了ControlNet如何控制单个神经网络块，论文中作者是以Stable Diffusion为例子，讲了如何使用ControlNet对大型网络进行控制。下图可以看到控制Stable Diffusion的过程就是将Encoder复制训练，decoder部分进行skip connection。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d3508466e1464a4eaf5000611ad6e491~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stable Diffusion有一个预处理步骤，将512×512的图片转化为64×64的图之后再进行训练，因此为了保证将控制条件也转化到64×64的条件空间上，训练过程中添加了一个小网络$E$将图像空间条件转化为特征图条件。  \n",
    "$$ c_{f} =E(c_{i} )$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个网络$E$是四层卷积神经网络，卷积核为4×4，步长为2，通道16，32，64，128，初始化为高斯权重。这个网络训练过程是和整个ControlNet进行联合训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练的目标函数为：  \n",
    "$$L=E_{z_{0} ,t,c_{t} ,c_{r} ,\\epsilon \\sim N(0,1)} [\\left \\| \\epsilon -\\epsilon _{\\theta }(z_{t} ,t,c_{t} ,c_{f} )  \\right \\| _{2}^{2} ] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用的就是人家Stable Diffusion原始的目标函数改了改。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "先看一下原始的Stable Diffusion的目标函数：  \n",
    "$$ L_{LDM} =E_{\\varepsilon (x) ,\\epsilon \\sim N(0,1),t} [\\left \\| \\epsilon -\\epsilon _{\\theta }(z_{t} ,t )  \\right \\| _{2}^{2} ]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将采样使用网络去噪之后和原图经过网络ϵ获得的潜变量计算loss，看其重建的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "那再回到:  \n",
    "$$L=E_{z_{0} ,t,c_{t} ,c_{r} ,\\epsilon \\sim N(0,1)} [\\left \\| \\epsilon -\\epsilon _{\\theta }(z_{t} ,t,c_{t} ,c_{f} )  \\right \\| _{2}^{2} ] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将原始图像经过$\\epsilon$之后获得潜变量，和经过网络$\\epsilon_\\theta$​重建之后的图算$L_2$ loss。原来Stable Diffusion中解码器要处理的是采样$z_t$​和时间步长$t$，在这里加了两个控制条件：\n",
    "- 文字prompt $c_{t} $\n",
    "- 任务相关的prompt $c_{f}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练过程中将50 %的文本提示$c_t$​随机替换为空字符串。这样有利于ControlNet网络从控制条件中识别语义内容。这样做的目的是，当Stable Diffusion没有prompt的时候，编码器能从输入的控制条件中获得更多的语义来代替prompt。（这也就是classifier-free guidance。）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 效果\n",
    "这一部分作者主要是讲了如何训练不同控制条件的ControlNet的，训练方法感兴趣的自己看，这里简单展示一下作者提供好的训练出来的模型。用《青蛇劫起》里边小青做一下示范："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Canny Edge\n",
    "使用Canny边缘检测生成边缘线稿，再将作为扩散模型输入。  \n",
    "![](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/fe094ba3485b44b4a711c90287934d56~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### HED\n",
    "使用hed边界检测。  \n",
    "![](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/ea3ace997d57427abcced5e0f3068391~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Depth\n",
    "使用深度图生成。  \n",
    "![](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/656af4c73c534622833ebc962d2fbcab~tplv-k3u1fbpfcp-zoom-in-crop-mark:1512:0:0:0.awebp?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 在SD Web UI中的应用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 安装ControlNET插件:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "地址：https://github.com/Mikubill/sd-webui-controlnet  \n",
    "找到右上角的扩展，然后点击从网址安装，把网址输入到下方  \n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-2f720fc228b9b68d8dc6e8a70f4dbaac_1440w.webp)\n",
    "\n",
    "下载完之后需要重新启动服务才可以生效（启动服务前下载）  \n",
    "成功标志：  \n",
    "\n",
    "![](https://pic2.zhimg.com/80/v2-1e9baa0676c87161415128489ab27335_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型下载\n",
    "ControlNET 相关模型（8个）  \n",
    "下载地址：https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main  \n",
    "再上面一行的选中 设置（settings），找到contronet ,调试多个contronet为3，最后重新载入UI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 插件controlnet介绍\n",
    "把模型放到刚下载的插件\\extensions\\sd-webui-controlnet\\models下，直接应用并重启界面  \n",
    "\n",
    "![](https://pic2.zhimg.com/80/v2-8551b69a7b81cfc1729009b4f4e3aadd_1440w.webp)  \n",
    "\n",
    "首页下方这块有ControlNET的模块，表示成功  \n",
    "\n",
    "![](https://pic4.zhimg.com/80/v2-2121f797bcf902c79da3aff7f739f3e7_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基础参数配置与介绍\n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-32b1809d0995d741310afa3edf3ce960_1440w.webp)  \n",
    "\n",
    "预处理器先选OpenPose，后面的模型一定跟预处理器对应一致，比如：预处理器我们选了OpenPose，模型选OpenPose命名的  \n",
    "\n",
    "![](https://pic4.zhimg.com/80/v2-8dcab890e51aa05fcb605146aeaddd87_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型介绍\n",
    "相应参考地址：https://link.zhihu.com/?target=http%3A//github.com/lllyasviel/ControlNet-v1-1-nightly  \n",
    "\n",
    "![](https://pic3.zhimg.com/80/v2-93c73ab9142e6eb9617eb71a6bb76b4a_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 Normal Map\n",
    "法线贴图控制稳定扩散  \n",
    "模型文件：control_v11p_sd15_normalbae.pth用途：用在模拟3D模型的表面细节和凹凸纹理  \n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-d3105572a02a7a6561eeb8a12a911920_1440w.webp)  \n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-f0f681d7ba580dfa8b5a3998e3117df4_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 Depth（MiDaS和LeReS深度信息估算）\n",
    "使用深度贴图控制稳定扩散。  \n",
    "模型文件：control_v11f1p_sd15_depth.pth;预处理器：Depth_Midas、Depth_Leres、Depth_Zoe  \n",
    "\n",
    "![](https://pic2.zhimg.com/80/v2-6ffd02f537047dad8fc1edf7096c0be9_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3 OpenPose\n",
    "使用Openpose控制稳定扩散。  \n",
    "模型文件：control_v11p_sd15_openpose.pth  \n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-2a03dba1cc17a06d820fe812899bda34_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4 MLSD\n",
    "用M-LSD直线控制稳定扩散。  \n",
    "模型文件：control_v11p_sd15_mlsd.pth适合建筑物和室内空间  \n",
    "\n",
    "![](https://pic3.zhimg.com/80/v2-f297e1ec10b3c423290892518d255f1e_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5 Canny\n",
    "用Canny映射控制稳定扩散。  \n",
    "模型文件：control_v11p_sd15_canny.pth  \n",
    "\n",
    "![](https://pic1.zhimg.com/80/v2-ebf8dcbc0ce37adb2ab629824088a984_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6 Hed\n",
    "用软边控制稳定扩散  \n",
    "模型文件：control_v11p_sd15_softedge.pth跟Canny基本相似，都是边缘检测提取线稿，唯一的不同是，Canny提取的线稿边更锐利，Hed的线稿边缘更柔结果更加接近原图  \n",
    "\n",
    "![](https://pic4.zhimg.com/80/v2-bce3ef6bbf5a510f79080bf4e893a04f_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7 scribble\n",
    "使用涂写控制稳定扩散模型文件：control_v11p_sd15_scribble.pth  \n",
    "\n",
    "![](https://pic2.zhimg.com/80/v2-2b82b345b4a400120823735c9f8c8841_1440w.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8 seg\n",
    "用语义分割控制稳定扩散。  \n",
    "模型文件：control_v11p_sd15_seg.pth  \n",
    "\n",
    "![](https://pic3.zhimg.com/80/v2-7aa11fee99d1dc4b3dd5435a5173ba1a_1440w.webp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dyh",
   "language": "python",
   "name": "dyh"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}