demo/denas/computer_vision/DENAS_CNN_DEMO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f3092e01",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/denas/computer_vision/DENAS_CNN_DEMO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6ce66fb",
   "metadata": {},
   "source": [
    "# AIOK DE-NAS for CNN Demo\n",
    "\n",
    "DE-NAS is a multi-model, hardware-aware, train-free NAS to construct compact model architectures for target platform directly. DE-NAS includes CNN-based search space for CV domain and Transformer-based search space for CV/NLP/ASR domains, and leverages hardware-aware train-free scoring method to evaluate the performance of the candidate architecture without training.\n",
    "\n",
    "This demo mainly introduces CV integration with DE-NAS to search lighter, faster, higher performance cnn-based and transformer-based ASR model in a training-free way."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8671985",
   "metadata": {},
   "source": [
    "# Content\n",
    "* [Overview](#Overview)\n",
    "    * [DE-NAS on CV(CNN) Domain](#DE-NAS-on-CV(CNN)-Domain)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "    * [3. Data Prepare](#3.-Data-Prepare)\n",
    "    * [4. Search](#4.-Search)\n",
    "    * [5. Train](#5.-Train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0c1e520",
   "metadata": {},
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "480c73af",
   "metadata": {},
   "source": [
    "## DE-NAS on CV(CNN) Domain\n",
    "\n",
    "For CNN models, the basic structure is constructure by Residual block or Bottleneck block, the Search Space for CNN domain as following:\n",
    "``` yaml\n",
    "SUPERNET structure:\n",
    "    \"SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)\n",
    "SuperResK3K3(16,32,2,16,1)SuperResK3K3(32,64,2,32,1)SuperResK3K3(64,64,2,32,1)SuperConvK1BNRELU(64,128,1,1)\"\n",
    "SEARCH SPACE:\n",
    "    number of layer:[18, 35, 50, 101]\n",
    "    Convolutional layer kernel size:[1x1, 3x3, 5x5, 7x7, 9x9, 11x11]\n",
    "    Number of filters:[8,16,32,64,128]\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d5408eb",
   "metadata": {},
   "source": [
    "## Performance\n",
    "\n",
    "<img src=\"./img/denas_cnn_stock.png\" width=\"400\"/>\n",
    "<img src=\"./img/denas_cnn_zennas.png\" width=\"400\" height=\"300\"/>\n",
    "\n",
    "* Testing methodology\n",
    "    * Dataset: CIFAR10, Metric: Top-1 accuracy 0.9579\n",
    "    * Baseline: ResNet50, Zen-NAS\n",
    "    * Training epoch: 200\n",
    "* DE-NAS CNN delivered 9.86x training speedup over ResNet50\n",
    "* DE-NAS CNN delivered 40.73x search and 82.57x training speedup over Zen-NAS with 39% model size reduction and 5% accuracy loss (0.91 vs. 0.9579)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5cc7ed3e",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "## 1. Environment Setup\n",
    "\n",
    "### (Option 1) Use Pip Install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "097ad85f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install e2eAIOK-denas --pre\n",
    "pip install torchsummary joblib"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "91c112dc",
   "metadata": {},
   "source": [
    "### (Option 2) Use Docker\n",
    "\n",
    "``` bash\n",
    "# Setup ENV\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "python3 scripts/start_e2eaiok_docker.py -b pytorch112 -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "# Enter Docker\n",
    "sshpass -p docker ssh ${host0} -p 12347\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8724642e",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "### Search Configuration\n",
    "\n",
    "``` yaml\n",
    "model_type: cnn\n",
    "search_engine: EvolutionarySearchEngine # Options: random search, Evolution algorithm, SigOpt search\n",
    "batch_size: 64\n",
    "random_max_epochs: 1000\n",
    "max_epochs: 1\n",
    "select_num: 50\n",
    "population_num: 50 # population number size for EA search\n",
    "crossover_num: 0\n",
    "mutation_num: 50 # mutation number size for EA search\n",
    "budget_num_layers: 18 # number of layer threshold\n",
    "budget_model_size: 1000000 # model parameter size threshold\n",
    "budget_flops: 10000000 # model FLOPs threshold\n",
    "img_size: 32 # input image size\n",
    "num_classes: 100\n",
    "plainnet_struct: \"SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK3K3(16,32,2,16,1)SuperResK3K3(32,64,2,32,1)SuperResK3K3(64,64,2,32,1)SuperConvK1BNRELU(64,128,1,1)\" # This is the supernet architechture\n",
    "no_reslink: False\n",
    "no_BN: False\n",
    "use_se: False\n",
    "seed: 0\n",
    "expressivity_weight: 1 # weight for train free score of expressivity \n",
    "complexity_weight: 0 # weight for train free score of complexity\n",
    "diversity_weight: 0 # weight for train free score of diversity score\n",
    "saliency_weight: 1 # weight for train free score of salience score\n",
    "latency_weight: 0 # weight for latency setup according to different platforms\n",
    "```\n",
    "\n",
    "### Training Configuration\n",
    "\n",
    "``` yaml\n",
    "domain: cnn\n",
    "train_epochs: 1\n",
    "eval_epochs: 1\n",
    "input_size: 32\n",
    "best_model_structure: /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/best_cnn_model_structure.txt\n",
    "num_classes: 10\n",
    "dist_backend: gloo\n",
    "train_batch_size: 128\n",
    "eval_batch_size: 128\n",
    "data_path: ~/data/pytorch_cifar10\n",
    "data_set: CIFAR10\n",
    "output_dir: ./\n",
    "num_workers: 10\n",
    "pin_mem: True\n",
    "eval_metric: \"accuracy\"\n",
    "learning_rate: 0.001\n",
    "momentum: 0.9\n",
    "weight_decay: 0.01\n",
    "optimizer: \"SGD\"\n",
    "criterion: \"CrossEntropyLoss\"\n",
    "lr_scheduler: \"CosineAnnealingLR\"\n",
    "print_freq: 10\n",
    "metric_threshold: 94\n",
    "mode: \"train\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f46f11b6",
   "metadata": {},
   "source": [
    "## 3. Data Prepare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b50c3cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "\n",
    "data_folder = \"~/data/pytorch_cifar\"\n",
    "is_train = True\n",
    "transform = None\n",
    "\n",
    "# Download Cifar10 Dataset\n",
    "dataset = datasets.CIFAR10(data_folder, train=is_train, transform=transform, download=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8721e28d",
   "metadata": {},
   "source": [
    "## 4. Search\n",
    "The input is the configuration for CNN domain, edit the configuration file of `e2eaiok_denas_cnn.conf` for CNN model search, and then run below commond line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2e95f9e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03/27/2023 01:37:08 - INFO - DENAS -   epoch = 0\n",
      "/opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/e2eAIOK/DeNas/scores/basic_utils.py:132: UserWarning: torch.symeig is deprecated in favor of torch.linalg.eigh and will be removed in a future PyTorch release.\n",
      "The default behavior has changed from using the upper triangular portion of the matrix by default to using the lower triangular portion.\n",
      "L, _ = torch.symeig(A, upper=upper)\n",
      "should be replaced with\n",
      "L = torch.linalg.eigvalsh(A, UPLO='U' if upper else 'L')\n",
      "and\n",
      "L, V = torch.symeig(A, eigenvectors=True)\n",
      "should be replaced with\n",
      "L, V = torch.linalg.eigh(A, UPLO='U' if upper else 'L') (Triggered internally at  ../aten/src/ATen/native/BatchLinearAlgebra.cpp:3041.)\n",
      "  eigenvalues, _ = torch.symeig(ntk)  # ascending\n",
      "03/27/2023 01:37:10 - INFO - DENAS -   random 1/1 structure SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK3K3(16,32,2,16,1)SuperResK3K3(32,64,2,32,1)SuperResK5K5(64,64,2,48,2)SuperConvK1BNRELU(64,128,1,1) nas_score -7.420200824737549 params 373164\n",
      "03/27/2023 01:37:10 - INFO - DENAS -   random_num = 1\n",
      "03/27/2023 01:37:12 - INFO - DENAS -   mutation 1/1 structure SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK1K3K1(16,40,2,16,1)SuperResK5K5(40,96,2,40,2)SuperResK3K3(96,64,2,32,1)SuperConvK1BNRELU(64,128,1,1) nas_score -5.963028430938721 params 416644\n",
      "03/27/2023 01:37:12 - INFO - DENAS -   mutation_num = 1\n",
      "03/27/2023 01:37:13 - INFO - DENAS -   crossover 1/1 structure SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK3K3(16,32,2,16,1)SuperResK3K3(32,64,2,32,1)SuperResK3K3(64,64,2,32,1)SuperConvK1BNRELU(64,128,1,1) nas_score -13.802251815795898 params 102572\n",
      "03/27/2023 01:37:13 - INFO - DENAS -   crossover_num = 1\n",
      "03/27/2023 01:37:13 - INFO - DENAS -   best structure SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK1K3K1(16,40,2,16,1)SuperResK5K5(40,96,2,40,2)SuperResK3K3(96,64,2,32,1)SuperConvK1BNRELU(64,128,1,1) nas_score -5.963028430938721 params 416644\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DE-NAS completed, best structure is SuperConvK3BNRELU(3,8,1,1)SuperResK3K3(8,16,1,8,1)SuperResK1K3K1(16,40,2,16,1)SuperResK5K5(40,96,2,40,2)SuperResK3K3(96,64,2,32,1)SuperConvK1BNRELU(64,128,1,1)\n"
     ]
    }
   ],
   "source": [
    "import yaml\n",
    "from easydict import EasyDict as edict\n",
    "from e2eAIOK.DeNas.cv.third_party.ZenNet import DeSearchSpaceXXBL, DeMainNet\n",
    "from e2eAIOK.DeNas.search.SearchEngineFactory import SearchEngineFactory\n",
    "\n",
    "# create common settings\n",
    "settings = {}\n",
    "settings[\"domain\"] = \"cnn\"\n",
    "# load search settings\n",
    "with open(\"/home/vmagent/app/e2eaiok/conf/denas/cv/e2eaiok_denas_cnn.conf\") as f:\n",
    "    conf = yaml.load(f, Loader=yaml.FullLoader)\n",
    "settings.update(conf)\n",
    "settings[\"max_epochs\"] = 1\n",
    "settings[\"population_num\"] = 1\n",
    "settings[\"crossover_num\"] = 1\n",
    "settings[\"mutation_num\"] = 1\n",
    "params = edict(settings)\n",
    "\n",
    "# create supernet and search space\n",
    "super_net = DeMainNet\n",
    "search_space = DeSearchSpaceXXBL\n",
    "\n",
    "# create search engine and launch search\n",
    "searcher = SearchEngineFactory.create_search_engine(params = params, super_net = super_net, search_space = search_space)\n",
    "searcher.search()\n",
    "# get best searched structure\n",
    "best_structure = searcher.get_best_structures()\n",
    "print(f\"DE-NAS completed, best structure is {best_structure}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5c44aa2e",
   "metadata": {},
   "source": [
    "## 5. Train\n",
    "The input is the configuration for CNN domain best model train, edit the configuration file of `e2eaiok_denas_train_cnn.conf` for CNN best model train, and then run below commond line\n",
    "> Note: Bellow training script is just for demonstration, and runs a small iterations. For actual performance result, please refer to [performance](#performance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2b992c19",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03/27/2023 01:50:13 - INFO - Trainer -   building model\n",
      "03/27/2023 01:50:13 - INFO - Trainer -   model created\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03/27/2023 01:50:14 - INFO - Trainer -   Trainer config: {'domain': 'cnn', 'train_epochs': 1, 'eval_epochs': 1, 'input_size': 32, 'best_model_structure': 'best_model_structure.txt', 'num_classes': 10, 'dist_backend': 'gloo', 'train_batch_size': 128, 'eval_batch_size': 128, 'data_path': '~/data/pytorch_cifar10', 'data_set': 'CIFAR10', 'output_dir': './', 'num_workers': 10, 'pin_mem': True, 'eval_metric': 'accuracy', 'learning_rate': 0.01, 'momentum': 0.9, 'weight_decay': 0.0005, 'optimizer': 'SGD', 'criterion': 'CrossEntropyLoss', 'lr_scheduler': 'CosineAnnealingLR', 'print_freq': 10, 'metric_threshold': 94, 'mode': 'train'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: [1]  [  0/391]  eta: 0:16:27  lr: 0.010000  loss: 2.3024 (2.3024)  time: 2.5243  data: 2.1748\n",
      "Epoch: [1]  [ 10/391]  eta: 0:01:47  lr: 0.010000  loss: 2.2757 (2.2825)  time: 0.2826  data: 0.1989\n",
      "Epoch: [1]  [ 20/391]  eta: 0:01:04  lr: 0.010000  loss: 2.2494 (2.2445)  time: 0.0570  data: 0.0011\n",
      "Epoch: [1]  [ 30/391]  eta: 0:00:49  lr: 0.010000  loss: 2.1454 (2.1947)  time: 0.0559  data: 0.0011\n",
      "Epoch: [1]  [ 40/391]  eta: 0:00:40  lr: 0.010000  loss: 2.0676 (2.1526)  time: 0.0554  data: 0.0013\n",
      "Epoch: [1]  [ 50/391]  eta: 0:00:35  lr: 0.010000  loss: 1.9885 (2.1133)  time: 0.0567  data: 0.0013\n",
      "Epoch: [1]  [ 60/391]  eta: 0:00:32  lr: 0.010000  loss: 1.9324 (2.0797)  time: 0.0578  data: 0.0013\n",
      "Epoch: [1]  [ 70/391]  eta: 0:00:29  lr: 0.010000  loss: 1.8920 (2.0458)  time: 0.0559  data: 0.0014\n",
      "Epoch: [1]  [ 80/391]  eta: 0:00:27  lr: 0.010000  loss: 1.8261 (2.0187)  time: 0.0561  data: 0.0014\n",
      "Epoch: [1]  [ 90/391]  eta: 0:00:25  lr: 0.010000  loss: 1.8120 (1.9946)  time: 0.0591  data: 0.0015\n",
      "Epoch: [1]  [100/391]  eta: 0:00:23  lr: 0.010000  loss: 1.7469 (1.9691)  time: 0.0624  data: 0.0015\n",
      "Epoch: [1]  [110/391]  eta: 0:00:22  lr: 0.010000  loss: 1.7173 (1.9476)  time: 0.0606  data: 0.0014\n",
      "Epoch: [1]  [120/391]  eta: 0:00:21  lr: 0.010000  loss: 1.6978 (1.9269)  time: 0.0584  data: 0.0013\n",
      "Epoch: [1]  [130/391]  eta: 0:00:19  lr: 0.010000  loss: 1.6744 (1.9085)  time: 0.0580  data: 0.0014\n",
      "Epoch: [1]  [140/391]  eta: 0:00:18  lr: 0.010000  loss: 1.6276 (1.8898)  time: 0.0563  data: 0.0014\n",
      "Epoch: [1]  [150/391]  eta: 0:00:17  lr: 0.010000  loss: 1.6336 (1.8743)  time: 0.0557  data: 0.0013\n",
      "Epoch: [1]  [160/391]  eta: 0:00:16  lr: 0.010000  loss: 1.6343 (1.8619)  time: 0.0564  data: 0.0013\n",
      "Epoch: [1]  [170/391]  eta: 0:00:15  lr: 0.010000  loss: 1.6165 (1.8443)  time: 0.0579  data: 0.0013\n",
      "Epoch: [1]  [180/391]  eta: 0:00:14  lr: 0.010000  loss: 1.5715 (1.8309)  time: 0.0573  data: 0.0013\n",
      "Epoch: [1]  [190/391]  eta: 0:00:14  lr: 0.010000  loss: 1.5864 (1.8183)  time: 0.0589  data: 0.0014\n",
      "Epoch: [1]  [200/391]  eta: 0:00:13  lr: 0.010000  loss: 1.5612 (1.8051)  time: 0.0595  data: 0.0014\n",
      "Epoch: [1]  [210/391]  eta: 0:00:12  lr: 0.010000  loss: 1.5459 (1.7924)  time: 0.0561  data: 0.0012\n",
      "Epoch: [1]  [220/391]  eta: 0:00:11  lr: 0.010000  loss: 1.5360 (1.7832)  time: 0.0556  data: 0.0011\n",
      "Epoch: [1]  [230/391]  eta: 0:00:10  lr: 0.010000  loss: 1.5481 (1.7735)  time: 0.0579  data: 0.0017\n",
      "Epoch: [1]  [240/391]  eta: 0:00:10  lr: 0.010000  loss: 1.5418 (1.7631)  time: 0.0587  data: 0.0018\n",
      "Epoch: [1]  [250/391]  eta: 0:00:09  lr: 0.010000  loss: 1.5418 (1.7542)  time: 0.0592  data: 0.0013\n",
      "Epoch: [1]  [260/391]  eta: 0:00:08  lr: 0.010000  loss: 1.4997 (1.7438)  time: 0.0602  data: 0.0013\n",
      "Epoch: [1]  [270/391]  eta: 0:00:08  lr: 0.010000  loss: 1.4873 (1.7356)  time: 0.0571  data: 0.0013\n",
      "Epoch: [1]  [280/391]  eta: 0:00:07  lr: 0.010000  loss: 1.4953 (1.7269)  time: 0.0544  data: 0.0013\n",
      "Epoch: [1]  [290/391]  eta: 0:00:06  lr: 0.010000  loss: 1.4633 (1.7176)  time: 0.0545  data: 0.0013\n",
      "Epoch: [1]  [300/391]  eta: 0:00:05  lr: 0.010000  loss: 1.4431 (1.7094)  time: 0.0545  data: 0.0013\n",
      "Epoch: [1]  [310/391]  eta: 0:00:05  lr: 0.010000  loss: 1.4425 (1.7004)  time: 0.0564  data: 0.0013\n",
      "Epoch: [1]  [320/391]  eta: 0:00:04  lr: 0.010000  loss: 1.4156 (1.6912)  time: 0.0581  data: 0.0013\n",
      "Epoch: [1]  [330/391]  eta: 0:00:03  lr: 0.010000  loss: 1.3811 (1.6823)  time: 0.0566  data: 0.0013\n",
      "Epoch: [1]  [340/391]  eta: 0:00:03  lr: 0.010000  loss: 1.3811 (1.6746)  time: 0.0567  data: 0.0013\n",
      "Epoch: [1]  [350/391]  eta: 0:00:02  lr: 0.010000  loss: 1.3713 (1.6650)  time: 0.0584  data: 0.0016\n",
      "Epoch: [1]  [360/391]  eta: 0:00:01  lr: 0.010000  loss: 1.3451 (1.6564)  time: 0.0572  data: 0.0014\n",
      "Epoch: [1]  [370/391]  eta: 0:00:01  lr: 0.010000  loss: 1.3765 (1.6496)  time: 0.0569  data: 0.0017\n",
      "Epoch: [1]  [380/391]  eta: 0:00:00  lr: 0.010000  loss: 1.3613 (1.6410)  time: 0.0563  data: 0.0019\n",
      "Epoch: [1]  [390/391]  eta: 0:00:00  lr: 0.010000  loss: 1.3571 (1.6339)  time: 0.0540  data: 0.0014\n",
      "Epoch: [1] Total time: 0:00:24 (0.0637 s / it)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/intel/oneapi/intelpython/latest/lib/python3.9/site-packages/torch/optim/lr_scheduler.py:156: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.\n",
      "  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test:[ 0/79]eta: 0:03:20loss: 1.3118 (1.3118)acc1: 52.3438 (52.3438)time: 2.5371data: 2.1760\n",
      "Test:[10/79]eta: 0:00:17loss: 1.3664 (1.3635)acc1: 51.5625 (50.7812)time: 0.2604data: 0.1996\n",
      "Test:[20/79]eta: 0:00:08loss: 1.3579 (1.3512)acc1: 50.0000 (50.3720)time: 0.0254data: 0.0013\n",
      "Test:[30/79]eta: 0:00:05loss: 1.3553 (1.3536)acc1: 50.0000 (50.3024)time: 0.0221data: 0.0047\n",
      "Test:[40/79]eta: 0:00:03loss: 1.3501 (1.3513)acc1: 50.7812 (50.7812)time: 0.0241data: 0.0068\n",
      "Test:[50/79]eta: 0:00:02loss: 1.3693 (1.3575)acc1: 50.0000 (50.6281)time: 0.0231data: 0.0052\n",
      "Test:[60/79]eta: 0:00:01loss: 1.3693 (1.3553)acc1: 50.7812 (50.8197)time: 0.0234data: 0.0057\n",
      "Test:[70/79]eta: 0:00:00loss: 1.3627 (1.3589)acc1: 51.5625 (50.7923)time: 0.0206data: 0.0038\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03/27/2023 01:50:43 - INFO - Trainer -   Evaluate time:4.431807279586792\n",
      "03/27/2023 01:50:43 - INFO - Trainer -   Epoch 1 training time:29.34877920150757\n",
      "03/27/2023 01:50:43 - INFO - Trainer -   Total time:29.349584817886353\n",
      "03/27/2023 01:50:43 - INFO - Trainer -   Trainer complete\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test:[78/79]eta: 0:00:00loss: 1.3352 (1.3558)acc1: 52.3438 (50.8999)time: 0.0199data: 0.0016\n",
      "Test: Total time: 0:00:04 (0.0560 s / it)\n",
      "* Acc@1 50.900 loss 1.356\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "50.89992088607595"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import yaml\n",
    "from easydict import EasyDict as edict\n",
    "import torch\n",
    "from e2eAIOK.DeNas.cv.model_builder_denas_cv import ModelBuilderCVDeNas\n",
    "from e2eAIOK.common.trainer.data.cv.data_builder_imagenet import DataBuilderImageNet\n",
    "from e2eAIOK.common.trainer.data.cv.data_builder_cifar import DataBuilderCIFAR\n",
    "import e2eAIOK.common.trainer.utils.utils as utils\n",
    "from e2eAIOK.DeNas.cv.cv_trainer import CVTrainer\n",
    "\n",
    "# create common settings\n",
    "settings = {}\n",
    "settings[\"domain\"] = \"cnn\"\n",
    "# load training settings\n",
    "with open(\"/home/vmagent/app/e2eaiok/conf/denas/cv/e2eaiok_denas_train_cnn.conf\") as f:\n",
    "    conf = yaml.load(f, Loader=yaml.FullLoader)\n",
    "settings.update(conf)\n",
    "settings[\"train_epochs\"] = 1\n",
    "settings[\"best_model_structure\"] = \"best_model_structure.txt\"\n",
    "cfg = edict(settings)\n",
    "\n",
    "# create CNN model builder and create CNN model\n",
    "model = ModelBuilderCVDeNas(cfg).create_model()\n",
    "# get training and evaluation dataloader\n",
    "train_dataloader, eval_dataloader = (DataBuilderImageNet(cfg) if cfg.data_set == 'ImageNet' else DataBuilderCIFAR(cfg)).get_dataloader()\n",
    "# create optimizer\n",
    "optimizer = utils.create_optimizer(model, cfg)\n",
    "criterion = utils.create_criterion(cfg)\n",
    "scheduler = utils.create_scheduler(optimizer, cfg)\n",
    "metric = utils.create_metric(cfg)\n",
    "# create CNN trainer\n",
    "trainer = CVTrainer(cfg, model, train_dataloader, eval_dataloader, optimizer, criterion, scheduler, metric)\n",
    "# start model training and evaluation\n",
    "trainer.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80e5feac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "960de6b0d90e8c1b5b9252244488628d965641900559c86d48ac5737e2de9daf"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}