load_project_iris.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63f13f12",
   "metadata": {},
   "source": [
    "# Load project YAML from - Git, Zip, Tar source"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1bd4e7d",
   "metadata": {},
   "source": [
    "After you create your project YAML you can simply load that project and run, build and deploy your function.\n",
    "You can found an addional information of how to create a project YAML for CI/CD [here](https://github.com/GiladShapira94/load-project-example/blob/master/CI-CD%20automation%20creation.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cebe620",
   "metadata": {},
   "source": [
    "In this session you would learn how to:\n",
    "* Load or Create a new project for remote URL\n",
    "* get a function object \n",
    "* run a function\n",
    "* build a function\n",
    "* deploy a function\n",
    "* run a workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "952f4638",
   "metadata": {},
   "source": [
    "Install mlrun - if does not install use ``pip install mlrun==<mlrun server version>`` or ``sh align_mlrun.sh`` (our default mlrun installer - automatically install the server version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6ed42495",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "48696a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "source = 'git://github.com/GiladShapira94/quick-start.git#master'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e1557",
   "metadata": {},
   "source": [
    "#### Load or Create a new project\n",
    "For loading a project you can use :\n",
    "1. [load_project](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=load_project#mlrun.projects.load_project) thats Load an MLRun project from git or tar or dir \n",
    "````\n",
    "# load the project and run the 'main' workflow\n",
    "project = load_project(\"./\", \"git://github.com/mlrun/project-demo.git\",url=<name (in DB) or git or tar.gz or .zip sources archive path>)\n",
    "````\n",
    "2. [new_project](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=new_project#mlrun.projects.new_project) thats Create a new MLRun project, optionally load it from a yaml/zip/git template.\n",
    "````\n",
    "# create a project with local and marketplace functions, a workflow, and an artifact\n",
    "project = mlrun.new_project(\"myproj\", \"./\", init_git=True, description=\"my new project\",url=<name (in DB) or git or tar.gz or .zip sources archive path>)\n",
    "````\n",
    "***Important Note-*** If you want to run your code locally you must clone the files to the context directory, execute automaticlly when you load project or create new project and define a url params\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "930b5fde",
   "metadata": {},
   "outputs": [],
   "source": [
    "project = mlrun.load_project('./clone',url=source,init_git=True,clone=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13538415",
   "metadata": {},
   "source": [
    "By Specifying a url/remote address MLRun automatically clone/extract your git repository ot tar/zip file\n",
    "\n",
    "if Clone = True, always clone **(delete any existing content)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "94bd3f1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind: project\n",
      "metadata:\n",
      "  name: quick-start-tutorial-shapira\n",
      "spec:\n",
      "  functions:\n",
      "  - name: prep_data\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: prep_data.prep_data\n",
      "    with_repo: true\n",
      "  - name: train_iris\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: train.train_iris\n",
      "    with_repo: true\n",
      "  - url: serving.yaml\n",
      "    name: serving\n",
      "    with_repo: true\n",
      "  workflows: []\n",
      "  artifacts: []\n",
      "  source: git://github.com/GiladShapira94/quick-start.git#refs/heads/master\n",
      "  origin_url: git://github.com/GiladShapira94/quick-start.git#refs/heads/master\n",
      "  load_source_on_run: true\n",
      "  desired_state: online\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(project.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1de659",
   "metadata": {},
   "source": [
    "#### Run Function \n",
    "For run a function you will need to use the [run_function](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=run_function#mlrun.projects.MlrunProject.run_function) method.\n",
    "This method allows you to run a MLRun **jobs** locally and remotely as long as there is no requirments ( if there is any requirments you will need to build a new image before you run a function)\n",
    "It is equivalent to func.run method.\n",
    "````\n",
    "project.run_function(function='<function name>',params={'num':3},local=False)\n",
    "````\n",
    "````\n",
    "project.run_function(function='<function name>',params={'num':3},local=True)\n",
    "````"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "da6a1c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_url = mlrun.get_sample_path(\"data/iris/iris.data.raw.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "618a5fa8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:06:54,504 [warning] run command, file or code were not specified\n",
      "> 2022-09-19 14:06:54,507 [info] starting run prep_data-prep_data uid=650f92a4ff9a4603b36c9eda066db032 DB=http://mlrun-api:8080\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>quick-start-tutorial-shapira</td>\n",
       "      <td><div title=\"650f92a4ff9a4603b36c9eda066db032\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/650f92a4ff9a4603b36c9eda066db032/overview\" target=\"_blank\" >...066db032</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 19 14:06:54</td>\n",
       "      <td>completed</td>\n",
       "      <td>prep_data-prep_data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=local</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">host=jupyter-shapira-6d47768cf9-jqk9v</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris.data.raw.csv\">source_url</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">num_rows=150</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulta43cf784\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/prep_data-prep_data/0/cleaned_data.csv\">cleaned_data</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulta43cf784-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulta43cf784-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulta43cf784\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulta43cf784-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/650f92a4ff9a4603b36c9eda066db032/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:06:55,358 [info] run executed, status=completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7efbecc72890>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.run_function(function='prep_data',inputs={'source_url': source_url},local=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3999db33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:06:55,366 [info] starting run prep_data-prep_data uid=52b933e5cd6a4391a42920d0d8db79d3 DB=http://mlrun-api:8080\n",
      "> 2022-09-19 14:06:55,664 [info] Job is running in the background, pod: prep-data-prep-data-mflfb\n",
      "> 2022-09-19 14:06:59,632 [info] extracting source from git://github.com/GiladShapira94/quick-start.git#refs/heads/master to /mlrun/code\n",
      "> 2022-09-19 14:07:00,037 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>quick-start-tutorial-shapira</td>\n",
       "      <td><div title=\"52b933e5cd6a4391a42920d0d8db79d3\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/52b933e5cd6a4391a42920d0d8db79d3/overview\" target=\"_blank\" >...d8db79d3</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 19 14:06:59</td>\n",
       "      <td>completed</td>\n",
       "      <td>prep_data-prep_data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">mlrun/client_version=1.1.0-rc22</div><div class=\"dictlist\">host=prep-data-prep-data-mflfb</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris.data.raw.csv\">source_url</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">num_rows=150</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result2db51b99\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/prep_data-prep_data/0/cleaned_data.csv\">cleaned_data</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result2db51b99-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result2db51b99-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result2db51b99\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result2db51b99-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/52b933e5cd6a4391a42920d0d8db79d3/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:07:02,048 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "prep_data_run=project.run_function(function='prep_data',inputs={'source_url': source_url},local=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08b7bcd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = 'v3io:///projects/quick-start-tutorial-shapira/artifacts/prep_data-prep_data/0/cleaned_data.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ea69429e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'v3io:///projects/quick-start-tutorial-shapira/artifacts/prep_data-prep_data/0/cleaned_data.csv'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "37080574",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:07:02,132 [info] starting run train_iris-train_iris uid=7a5d231330a4457a899db730c0611bf5 DB=http://mlrun-api:8080\n",
      "> 2022-09-19 14:07:02,402 [info] Job is running in the background, pod: train-iris-train-iris-bbtxj\n",
      "> 2022-09-19 14:07:06,274 [info] extracting source from git://github.com/GiladShapira94/quick-start.git#refs/heads/master to /mlrun/code\n",
      "> 2022-09-19 14:07:08,679 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>quick-start-tutorial-shapira</td>\n",
       "      <td><div title=\"7a5d231330a4457a899db730c0611bf5\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/7a5d231330a4457a899db730c0611bf5/overview\" target=\"_blank\" >...c0611bf5</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Sep 19 14:07:06</td>\n",
       "      <td>completed</td>\n",
       "      <td>train_iris-train_iris</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">mlrun/client_version=1.1.0-rc22</div><div class=\"dictlist\">host=train-iris-train-iris-bbtxj</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte127b668\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/prep_data-prep_data/0/cleaned_data.csv\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div></td>\n",
       "      <td><div class=\"dictlist\">accuracy=1.0</div><div class=\"dictlist\">f1_score=1.0</div><div class=\"dictlist\">precision_score=1.0</div><div class=\"dictlist\">recall_score=1.0</div><div class=\"dictlist\">auc-micro=1.0</div><div class=\"dictlist\">auc-macro=1.0</div><div class=\"dictlist\">auc-weighted=1.0</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte127b668\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/train_iris-train_iris/0/feature-importance.html\">feature-importance</div><div title=\"v3io:///projects/quick-start-tutorial-shapira/artifacts/train_iris-train_iris/0/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte127b668\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/train_iris-train_iris/0/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte127b668\" title=\"files/v3io/projects/quick-start-tutorial-shapira/artifacts/train_iris-train_iris/0/roc-curves.html\">roc-curves</div><div title=\"v3io:///projects/quick-start-tutorial-shapira/artifacts/train_iris-train_iris/0/model/\">model</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulte127b668-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulte127b668-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulte127b668\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulte127b668-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/mlprojects/quick-start-tutorial-shapira/jobs/monitor/7a5d231330a4457a899db730c0611bf5/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:07:11,882 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "train_run=project.run_function(function='train_iris',inputs={'dataset': dataset},params={'label_column': 'label'},local=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735b50c2",
   "metadata": {},
   "source": [
    "#### Get Function Object \n",
    "For gets function object you will need to use the [get_function](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=get_function#mlrun.projects.MlrunProject.get_function) method.\n",
    "This method allows you to get a function object based on the metadata in your project YAML file.\n",
    "````\n",
    "serving_func = project.get_function('<function name>')\n",
    "````"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08622555",
   "metadata": {},
   "source": [
    "Test your serving funciton locally before deploying it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb298fbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "serving_func=project.get_function('serving')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bf767584",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_data = '''{\"inputs\":[[5.1, 3.5, 1.4, 0.2],[7.7, 3.8, 6.7, 2.2]]}'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a3c609f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:09:15,856 [info] model model was loaded\n",
      "> 2022-09-19 14:09:15,857 [info] Loaded ['model']\n"
     ]
    }
   ],
   "source": [
    "serving_server = serving_func.to_mock_server()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "773b71a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "X does not have valid feature names, but RandomForestClassifier was fitted with feature names\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'id': '40cb10e3839a4513b8c8f33bfc73319f',\n",
       " 'model_name': 'model',\n",
       " 'outputs': [0, 2]}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_server.test('/',my_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8f25df",
   "metadata": {},
   "source": [
    "**Changed pull_at_runtime to False for pull files on image creation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e4852481",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind: project\n",
      "metadata:\n",
      "  name: quick-start-tutorial-shapira\n",
      "spec:\n",
      "  functions:\n",
      "  - name: prep_data\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: prep_data.prep_data\n",
      "    with_repo: true\n",
      "  - name: train_iris\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: train.train_iris\n",
      "    with_repo: true\n",
      "  - url: serving.yaml\n",
      "    name: serving\n",
      "    with_repo: true\n",
      "  workflows: []\n",
      "  artifacts: []\n",
      "  source: git://github.com/GiladShapira94/quick-start.git#master\n",
      "  origin_url: git://github.com/GiladShapira94/quick-start.git#refs/heads/master\n",
      "  load_source_on_run: false\n",
      "  desired_state: online\n",
      "\n"
     ]
    }
   ],
   "source": [
    "project.set_source(source,pull_at_runtime=False)\n",
    "print(project.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e01284c",
   "metadata": {},
   "source": [
    "#### Deploy Function\n",
    "For deplying remote function as nuclio or serving you will need to use the [deploy_function](https://docs.mlrun.org/en/latest/api/mlrun.projects.html?highlight=deploy_function#mlrun.projects.MlrunProject.deploy_function) method.\n",
    "You must use this method before invoke nuclio or serving fucntions.\n",
    "It is equivalent to func.deploy() method.\n",
    "````\n",
    "nuclio_func=project.deploy_function(function='<function name>')\n",
    "\n",
    "nuclio_func.function.invoke('/',{'int':4})\n",
    "````"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ac6e6e41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:09:23,356 [info] Starting remote function deploy\n",
      "2022-09-19 14:09:23  (info) Deploying function\n",
      "2022-09-19 14:09:23  (info) Building\n",
      "2022-09-19 14:09:23  (info) Staging files and preparing base images\n",
      "2022-09-19 14:09:24  (info) Building processor image\n",
      "2022-09-19 14:10:09  (info) Build complete\n",
      "2022-09-19 14:10:37  (info) Function deploy complete\n",
      "> 2022-09-19 14:10:38,294 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-quick-start-tutorial-shapira-serving.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['quick-start-tutorial-shapira-serving-quick-start-tutor-41yvaye0.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/']}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://quick-start-tutorial-shapira-serving-quick-start-tutor-41yvaye0.default-tenant.app.cust-cs-il-3-4-1.iguazio-cd1.com/', 'name': 'quick-start-tutorial-shapira-serving'})"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.deploy_function('serving')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e56db968",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-09-19 14:11:39,815 [info] invoking function: {'method': 'POST', 'path': 'http://nuclio-quick-start-tutorial-shapira-serving.default-tenant.svc.cluster.local:8080/'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'id': '1fb5a6bd-da6a-4b17-827f-55e7861cc816',\n",
       " 'model_name': 'model',\n",
       " 'outputs': [0, 2]}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_func.invoke('/',my_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38c9ab9c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}