Merge branch 'development' of https://github.com/mlrun/demos

README.md

@@ -4,13 +4,16 @@ The mlrun/demos repository provides full end-to-end ML demo use-case application
 
 #### In This Document
 
-- [Overview](#overview)
-- [Prerequisites](#prerequisites)
-- [scikit-learn demo](#demo-sklearn-pipe) — full AutoML pipeline (Iris data set)
-- [Horovod demo](#demo-horovd-image-classification) — image classification with distributed training
-- [Faces demo](#demo-face-recognition)— real-time image recognition, classification, and tracking with deep learning
-- [Churn demo](#demo-churn)— real-time customer-churn prediction (Telco Customer Churn data set)
-- [NetOps demo](#demo-netops)— predictive network operations/telemetry
+- [MLRun Demos](#mlrun-demos)
+      - [In This Document](#in-this-document)
+  - [Overview](#overview)
+    - [General ML Workflow](#general-ml-workflow)
+  - [Prerequisites](#prerequisites)
+  - [scikit-learn Demo: Full AutoML Pipeline](#scikit-learn-demo-full-automl-pipeline)
+  - [Horovod Demo: Image Classification with Distributed Training](#horovod-demo-image-classification-with-distributed-training)
+  - [Faces Demo: Real-Time Image Recognition with Deep Learning](#faces-demo-real-time-image-recognition-with-deep-learning)
+  - [Churn Demo: Real-Time Customer-Churn Prediction](#churn-demo-real-time-customer-churn-prediction)
+  - [NetOps Demo: Predictive Network Operations/Telemetry](#netops-demo-predictive-network-operationstelemetry)
 
 <a id="overview"></a>
 ## Overview
@@ -43,10 +46,10 @@ To run the MLRun demos, first do the following:
   See the instructions in the [MLRun documentation](https://github.com/mlrun/mlrun/blob/master/README.md#installation).
 - Ensure that your cluster has a shared file or object storage for storing the data (artifacts).
 
-<a id="demo-sklearn-pipe"></a>
+<a id="demo-scikit-learn-pipeline"></a>
 ## scikit-learn Demo: Full AutoML Pipeline
 
-The [**sklearn-pipe**](./sklearn-pipe/README.md) demo demonstrates how to build a full end-to-end automated-ML (AutoML) pipeline using [scikit-learn](https://scikit-learn.org) and the UCI [Iris data set](http://archive.ics.uci.edu/ml/datasets/iris).
+The [**scikit-learn-pipeline**](./scikit-learn-pipeline/README.md) demo demonstrates how to build a full end-to-end automated-ML (AutoML) pipeline using [scikit-learn](https://scikit-learn.org) and the UCI [Iris data set](http://archive.ics.uci.edu/ml/datasets/iris).
 
 The combined CI/data/ML pipeline includes the following steps:
 
@@ -57,21 +60,21 @@ The combined CI/data/ML pipeline includes the following steps:
 - Deploy the model as a real-time serverless function.
 - Test the serverless function's REST API with a test data set.
 
-To run the demo, download the [**sklearn-project.ipynb**](./sklearn-pipe/sklearn-project.ipynb) notebook into an empty directory and execute the cells sequentially.
+To run the demo, download the [**sklearn-project.ipynb**](./scikit-learn-pipeline/sklearn-project.ipynb) notebook into an empty directory and execute the cells sequentially.
 
 <br><p align="center"><img src="./docs/trees.png" width="500"/></p><br>
 
-<a id="demo-sklearn-pipe-pipeline-output"></a>
+<a id="demo-scikit-learn-pipeline-pipeline-output"></a>
 **Pipeline Output**
 
-The output plots can be viewed as static HTML files in the [sklearn-pipe/plots](sklearn-pipe/plots) directory.
+The output plots can be viewed as static HTML files in the [scikit-learn-pipeline/plots](scikit-learn-pipeline/plots) directory.
 
 <br><p align="center"><img src="./docs/skpipe.png" width="500"/></p><br>
 
 <a id="demo-horovd-image-classification"></a>
 ## Horovod Demo: Image Classification with Distributed Training
 
-The [**horovod-pipe**](horovod-pipe/README.md) demo demonstrates an end-to-end image-classification solution using [TensorFlow](https://www.tensorflow.org/) (versions 1 or 2), [Keras](https://keras.io/), [Horovod](https://eng.uber.com/horovod/), and [Nuclio](https://nuclio.io/).
+The [**image-classification-with-distributed-training**](image-classification-with-distributed-training/README.md) demo demonstrates an end-to-end image-classification solution using [TensorFlow](https://www.tensorflow.org/) (versions 1 or 2), [Keras](https://keras.io/), [Horovod](https://eng.uber.com/horovod/), and [Nuclio](https://nuclio.io/).
 
 The demo consists of four MLRun and Nuclio functions and a Kubeflow Pipelines orchestration:
 
@@ -93,7 +96,7 @@ The demo consists of four MLRun and Nuclio functions and a Kubeflow Pipelines or
 <a id="demo-face-recognition"></a>
 ## Faces Demo: Real-Time Image Recognition with Deep Learning
 
-The [**faces**](faces/README.md) demo demonstrates real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities.
+The [**faces**](realtime-face-recognition/README.md) demo demonstrates real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities.
 
 This comprehensive demonstration includes multiple components:
 
@@ -106,12 +109,12 @@ This comprehensive demonstration includes multiple components:
 <a id="demo-face-recognition-pipeline-output"></a>
 **Pipeline Output**
 
-<br><p align="center"><img src="./faces/workflow.png" width="500"/></p><br>
+<br><p align="center"><img src="./realtime-face-recognition/workflow.png" width="500"/></p><br>
 
 <a id="demo-churn"></a>
 ## Churn Demo: Real-Time Customer-Churn Prediction
 
-The [**chrun**](./churn/README.md) demo demonstrates analyses of customer-churn data using the Kaggle [Telco Customer Churn data set](https://www.kaggle.com/blastchar/telco-customer-churn), model training and validation using [XGBoost](https://xgboost.readthedocs.io), and model serving using real-time Nuclio serverless functions.
+The [**chrun**](./customer-churn-prediction/README.md) demo demonstrates analyses of customer-churn data using the Kaggle [Telco Customer Churn data set](https://www.kaggle.com/blastchar/telco-customer-churn), model training and validation using [XGBoost](https://xgboost.readthedocs.io), and model serving using real-time Nuclio serverless functions.
 
 The demo consists of few MLRun and Nuclio functions and a Kubeflow Pipelines orchestration:
 
@@ -125,14 +128,14 @@ The demo consists of few MLRun and Nuclio functions and a Kubeflow Pipelines orc
 <a id="demo-churn-pipeline-output"></a>
 **Pipeline Output**
 
-<br><p align="center"><img src="./churn/assets/pipeline-3.png" width="500"/></p><br>
+<br><p align="center"><img src="./customer-churn-prediction/assets/pipeline-3.png" width="500"/></p><br>
 
 <a id="demo-netops"></a>
 ## NetOps Demo: Predictive Network Operations/Telemetry
 <!-- TODO: If and when the demo is moved to the mlrun/demos repo, update the
   README link below. -->
 
-The [NetOps demo](https://github.com/mlrun/demo-network-operations/blob/master/README.md) demonstrates ingestion of telemetry/Network Operations (NetOps) data from a simulator or live stream, feature exploration, data preparation (aggregation), model training, and automated model deployment.
+The [NetOps demo](network-operations/README.md) demonstrates ingestion of telemetry/Network Operations (NetOps) data from a simulator or live stream, feature exploration, data preparation (aggregation), model training, and automated model deployment.
 
 The demo is maintained in a separate Git repository and also demonstrates how to manage a project life cycle using Git.
 

churn/churn-project.ipynb → customer-churn-prediction/churn-project.ipynb

@@ -228,7 +228,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "DATA_URL = \"https://raw.githubusercontent.com/mlrun/demos/master/churn/WA_Fn-UseC_-Telco-Customer-Churn.csv\"\n",
+    "DATA_URL = \"https://raw.githubusercontent.com/mlrun/demos/master/customer-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv\"\n",
     "\n",
     "churn_proj.log_artifact(\"raw-data\", target_path=DATA_URL)"
    ]
@@ -438,7 +438,7 @@
        "      <td><div title=\"store:///raw-data\">src</div></td>\n",
        "      <td><div class=\"dictlist\">file_ext=csv</div><div class=\"dictlist\">apply_tenure_map=False</div></td>\n",
        "      <td></td>\n",
-       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/churn/data/models/encoders/preproc-column_map.json\">preproc-column_map.json</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/churn/data/models/encoders/preproc-numcat_map.json\">preproc-numcat_map.json</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/churn/data/cleaned-data.csv\">cleaned-data</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/churn/data/encoded-data.csv\">encoded-data</div><div title=\"/User/ml/demos/churn/data/models/encoders\">model</div></td>\n",
+       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/customer-churn-prediction/data/models/encoders/preproc-column_map.json\">preproc-column_map.json</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/customer-churn-prediction/data/models/encoders/preproc-numcat_map.json\">preproc-numcat_map.json</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/customer-churn-prediction/data/cleaned-data.csv\">cleaned-data</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result649f10d5\" title=\"/files/ml/demos/customer-churn-prediction/data/encoded-data.csv\">encoded-data</div><div title=\"/User/ml/demos/customer-churn-prediction/data/models/encoders\">model</div></td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
@@ -616,7 +616,7 @@
     "    for f in functions.values():\n",
     "        f.apply(mount_v3io())\n",
     "        \n",
-    "    functions[\"server\"].set_env(\"INFERENCE_STREAM\", \"users/admin/artifacts/churn/model_stream\")\n",
+    "    functions[\"server\"].set_env(\"INFERENCE_STREAM\", \"users/admin/artifacts/customer-churn-prediction/model_stream\")\n",
     "\n",
     "    \n",
     "@dsl.pipeline(\n",
@@ -696,14 +696,14 @@
     "    test_xgb = funcs[\"xgbtest\"].as_step(\n",
     "        name=\"test-classifier\",\n",
     "        params={\"label_column\": \"labels\",\n",
-    "                \"plots_dest\"  : \"churn/test/xgb\"},\n",
+    "                \"plots_dest\"  : \"customer-churn-prediction/test/xgb\"},\n",
     "        inputs={\"models_path\" : xgb.outputs[\"model\"],\n",
     "                \"test_set\"    : xgb.outputs[\"test_set\"]})\n",
     "\n",
     "    test_cox = funcs[\"coxtest\"].as_step(\n",
     "        name=\"test-regressor\",\n",
     "        params={\"label_column\": \"labels\",\n",
-    "                \"plots_dest\"  : \"churn/test/cox\"},\n",
+    "                \"plots_dest\"  : \"customer-churn-prediction/test/cox\"},\n",
     "        inputs={\"models_path\" : cox.outputs[\"cx-model\"],\n",
     "                \"test_set\"    : cox.outputs[\"tenured-test-set\"]})\n",
     "\n",

churn/project/project.yaml → customer-churn-prediction/project/project.yaml

@@ -88,7 +88,7 @@ workflows:
     \ False\n\n# init functions is used to configure function resources and local\
     \ settings\ndef init_functions(functions: dict, project=None, secrets=None):\n\
     \    for f in functions.values():\n        f.apply(mount_v3io())\n        \n \
-    \   functions[\"server\"].set_env(\"INFERENCE_STREAM\", \"users/admin/artifacts/churn/model_stream\"\
+    \   functions[\"server\"].set_env(\"INFERENCE_STREAM\", \"users/admin/artifacts/customer-churn-prediction/model_stream\"\
     )\n\n    \n@dsl.pipeline(\n    name=\"Demo training pipeline\",\n    description=\"\
     Shows how to use mlrun.\"\n)\ndef kfpipeline():\n    \n    # encode the data\n\
     \    clean = funcs[\"clean-data\"].as_step(\n        name=\"clean-data\",\n  \
@@ -133,11 +133,11 @@ workflows:
     csv\"\n               },\n        inputs={\"dataset\"   : clean.outputs[\"encoded-data\"\
     ]},\n        outputs=[\"cx-model\", \"tenured-test-set\"])\n\n    test_xgb = funcs[\"\
     xgbtest\"].as_step(\n        name=\"test-classifier\",\n        params={\"label_column\"\
-    : \"labels\",\n                \"plots_dest\"  : \"churn/test/xgb\"},\n      \
+    : \"labels\",\n                \"plots_dest\"  : \"customer-churn-prediction/test/xgb\"},\n      \
     \  inputs={\"models_path\" : xgb.outputs[\"model\"],\n                \"test_set\"\
     \    : xgb.outputs[\"test_set\"]})\n\n    test_cox = funcs[\"coxtest\"].as_step(\n\
     \        name=\"test-regressor\",\n        params={\"label_column\": \"labels\"\
-    ,\n                \"plots_dest\"  : \"churn/test/cox\"},\n        inputs={\"\
+    ,\n                \"plots_dest\"  : \"customer-churn-prediction/test/cox\"},\n        inputs={\"\
     models_path\" : cox.outputs[\"cx-model\"],\n                \"test_set\"    :\
     \ cox.outputs[\"tenured-test-set\"]})\n\n    # deploy our model as a serverless\
     \ function\n    deploy_xgb = funcs[\"server\"].deploy_step(\n        models={\"\
@@ -147,5 +147,5 @@ artifacts:
   kind: ''
   iter: 0
   tree: latest
-  target_path: https://raw.githubusercontent.com/mlrun/demos/master/churn/WA_Fn-UseC_-Telco-Customer-Churn.csv
+  target_path: https://raw.githubusercontent.com/mlrun/demos/master/customer-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv
   db_key: raw-data

churn/project/workflow.py → customer-churn-prediction/project/workflow.py

@@ -10,7 +10,7 @@ def init_functions(functions: dict, project=None, secrets=None):
     for f in functions.values():
         f.apply(mount_v3io())
 
-    functions["server"].set_env("INFERENCE_STREAM", "users/admin/artifacts/churn/model_stream")
+    functions["server"].set_env("INFERENCE_STREAM", "users/admin/artifacts/customer-churn-prediction/model_stream")
 
 
 @dsl.pipeline(
@@ -90,14 +90,14 @@ def kfpipeline():
     test_xgb = funcs["xgbtest"].as_step(
         name="test-classifier",
         params={"label_column": "labels",
-                "plots_dest"  : "churn/test/xgb"},
+                "plots_dest"  : "customer-churn-prediction/test/xgb"},
         inputs={"models_path" : xgb.outputs["model"],
                 "test_set"    : xgb.outputs["test_set"]})
 
     test_cox = funcs["coxtest"].as_step(
         name="test-regressor",
         params={"label_column": "labels",
-                "plots_dest"  : "churn/test/cox"},
+                "plots_dest"  : "customer-churn-prediction/test/cox"},
         inputs={"models_path" : cox.outputs["cx-model"],
                 "test_set"    : cox.outputs["tenured-test-set"]})
 

horovod-pipe/README.md → image-classification-with-distributed-training/README.md

@@ -12,10 +12,10 @@ The demo consists of four MLRun and Nuclio functions and a Kubeflow Pipelines or
 > **Note:** The demo supports both TensorFlow versions 1 and 2.
 > There's one shared notebook and two code files — one for each TensorFlow version.
 
-<br><p align="center"><img src="/docs/hvd-flow.png" width="600"/></p><br>
+<br><p align="center"><img src="../docs/hvd-flow.png" width="600"/></p><br>
 
 <a id="pipeline-output"></a>
 ## Pipeline Output
 
-<br><p align="center"><img src="/docs/hvd-pipe.png" width="500"/></p><br>
+<br><p align="center"><img src="../docs/hvd-pipe.png" width="500"/></p><br>
 

horovod-pipe/horovod-project.ipynb → image-classification-with-distributed-training/horovod-project.ipynb

@@ -196,7 +196,7 @@
     {
      "data": {
       "text/plain": [
-       "('/User/mlrun-demos/demos/horovod-pipe/src-tfv2', '/User/artifacts/images')"
+       "('/User/mlrun-demos/demos/image-classification-with-distributed-training/src-tfv2', '/User/artifacts/images')"
       ]
      },
      "execution_count": 8,
@@ -1134,23 +1134,23 @@
        "      <td>completed</td>\n",
        "      <td>train</td>\n",
        "      <td><div class=\"dictlist\">loss=0.4674202799797058</div><div class=\"dictlist\">accuracy=0.808407723903656</div></td>\n",
-       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels/training-summary.html\">summary.html</div><div title=\"/User/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels/model-weights.h5\">model-weights</div><div title=\"/User/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels\">model</div></td>\n",
+       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels/training-summary.html\">summary.html</div><div title=\"/User/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels/model-weights.h5\">model-weights</div><div title=\"/User/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/tfmodels\">model</div></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><div title=\"744ccc09fb8e4d19a5353bffb815a5f6\"><a href=\"https://mlrun-ui.default-tenant.app.dev6.lab.iguazeng.com/projects/cat-and-dog-servers/jobs/744ccc09fb8e4d19a5353bffb815a5f6/info\" target=\"_blank\" >...b815a5f6</a></div></td>\n",
        "      <td>Oct 19 11:23:34</td>\n",
        "      <td>completed</td>\n",
        "      <td>label</td>\n",
        "      <td></td>\n",
-       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/categories_map.json\">categories_map</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/file_categories_df.csv\">file_categories</div></td>\n",
+       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/categories_map.json\">categories_map</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/file_categories_df.csv\">file_categories</div></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td><div title=\"a5329e3afc6b4d3ebbe411aef57d1e3f\"><a href=\"https://mlrun-ui.default-tenant.app.dev6.lab.iguazeng.com/projects/cat-and-dog-servers/jobs/a5329e3afc6b4d3ebbe411aef57d1e3f/info\" target=\"_blank\" >...f57d1e3f</a></div></td>\n",
        "      <td>Oct 19 11:23:02</td>\n",
        "      <td>completed</td>\n",
        "      <td>download</td>\n",
        "      <td></td>\n",
-       "      <td><div title=\"/User/mlrun-demos/demos/horovod-pipe/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/images\">content</div></td>\n",
+       "      <td><div title=\"/User/mlrun-demos/demos/image-classification-with-distributed-training/pipe/2cb80d4b-0cec-4139-8a47-fb63d7b14645/images\">content</div></td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>"

horovod-pipe/project.yaml → image-classification-with-distributed-training/project.yaml

@@ -73,7 +73,7 @@ functions:
       project: cat-and-dog-servers
       categories: []
     spec:
-      command: /User/mlrun-demos/demos/horovod-pipe/src-tfv2/horovod-training.py
+      command: /User/mlrun-demos/demos/image-classification-with-distributed-training/src-tfv2/horovod-training.py
       args: []
       image: mlrun/ml-models-gpu
       env: []