sdk/python/jobs/automl-standalone-jobs/automl-forecasting-orange-juice-sales/automl-forecasting-orange-juice-sales-mlflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML: Train \"the best\" Time-Series Forecasting model for the Orange Juice Sales Dataset.\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace. [Check this notebook for creating a workspace](../../../resources/workspace/workspace.ipynb)\n",
    "- A Compute Cluster. [Check this notebook to create a compute cluster](../../../resources/compute/compute.ipynb)\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Connect to your AML workspace from the Python SDK\n",
    "- Create an `AutoML time-series forecasting Job` with the 'forecasting()' factory-fuction.\n",
    "- Train the model using AmlCompute by submitting/running the AutoML forecasting training job\n",
    "- Obtaing the model and score predictions with it\n",
    "\n",
    "**Motivations** - This notebook explains how to setup and run an AutoML forecasting job. This is one of the nine ML-tasks supported by AutoML. Other ML-tasks are 'regression', 'classification', 'image classification', 'image object detection', 'nlp text classification', etc.\n",
    "\n",
    "In this example we use the associated Orange Juice Sales dataset to showcase how you can use AutoML for a simple forecasting problem and explore the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634852261599
    }
   },
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml import MLClient\n",
    "\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "from azure.ai.ml import automl\n",
    "from azure.ai.ml import Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634852261884
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "credential = DefaultAzureCredential()\n",
    "ml_client = None\n",
    "try:\n",
    "    ml_client = MLClient.from_config(credential)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your AML workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "\n",
    "    ml_client = MLClient(credential, subscription_id, resource_group, workspace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Azure ML Workspace information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "workspace = ml_client.workspaces.get(name=ml_client.workspace_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Workspace\"] = ml_client.workspace_name\n",
    "output[\"Subscription ID\"] = ml_client.connections._subscription_id\n",
    "output[\"Resource Group\"] = workspace.resource_group\n",
    "output[\"Location\"] = workspace.location\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.Data\n",
    "You are now ready to load the historical orange juice sales data.\n",
    "\n",
    "We will load the data into DataFrame objects, split the data to train and test datasets, creating the Azure Machine Learning MLTable objects to prepare for the later training and inference steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Load the data file into DataFrame.\n",
    "We will load the CSV file into a plain pandas DataFrame; the time column in the CSV is called _WeekStarting_, so it will be specially parsed into the datetime type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_column_name = \"WeekStarting\"\n",
    "data = pd.read_csv(\"./data/dominicks_OJ.csv\", parse_dates=[time_column_name])\n",
    "\n",
    "# Drop the columns 'logQuantity' as it is a leaky feature.\n",
    "data.drop(\"logQuantity\", axis=1, inplace=True)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row in the DataFrame holds a quantity of weekly sales for an OJ brand at a single store. The data also includes the sales price, a flag indicating if the OJ brand was advertised in the store that week, and some customer demographic information based on the store location. For historical reasons, the data also include the logarithm of the sales quantity. The Dominick's grocery data is commonly used to illustrate econometric modeling techniques where logarithms of quantities are generally preferred.\n",
    "\n",
    "The task is now to build a time-series model for the _Quantity_ column. It is important to note that this dataset is comprised of many individual time-series - one for each unique combination of _Store_ and _Brand_. To distinguish the individual time-series, we define the **time_series_id_column_names** - the columns whose values determine the boundaries between time-series: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_series_id_column_names = [\"Store\", \"Brand\"]\n",
    "nseries = data.groupby(time_series_id_column_names).ngroups\n",
    "print(\"Data contains {0} individual time-series.\".format(nseries))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For demonstration purposes, we extract sales time-series for just a few of the stores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_stores = [2, 5, 8]\n",
    "data_subset = data[data.Store.isin(use_stores)]\n",
    "nseries = data_subset.groupby(time_series_id_column_names).ngroups\n",
    "print(\"Data subset contains {0} individual time-series.\".format(nseries))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Data Splitting\n",
    "We now split the data into a training and a testing set for later forecast evaluation. The test set will contain the final 20 weeks of observed sales for each time-series. The splits should be stratified by series, so we use a group-by statement on the time series identifier columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_test_periods = 20\n",
    "\n",
    "\n",
    "def split_last_n_by_series_id(df, n):\n",
    "    \"\"\"Group df by series identifiers and split on last n rows for each group.\"\"\"\n",
    "    df_grouped = df.sort_values(time_column_name).groupby(  # Sort by ascending time\n",
    "        time_series_id_column_names, group_keys=False\n",
    "    )\n",
    "    df_head = df_grouped.apply(lambda dfg: dfg.iloc[:-n])\n",
    "    df_tail = df_grouped.apply(lambda dfg: dfg.iloc[-n:])\n",
    "    return df_head, df_tail\n",
    "\n",
    "\n",
    "train, test = split_last_n_by_series_id(data_subset, n_test_periods)\n",
    "\n",
    "# Save the DataFrame objects to files\n",
    "train_data_path = \"./data/dominicks_OJ_train.csv\"\n",
    "test_data_path = \"./data/dominicks_OJ_test.csv\"\n",
    "train.to_csv(train_data_path, index=False)\n",
    "test.to_csv(test_data_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Create the Azure Machine Learning MLTable\n",
    "\n",
    "With Azure Machine Learning MLTable you can keep a single copy of data in your storage, easily access data during model training, share data and collaborate with other users.\n",
    "Below, we will upload the data by creating an MLTable to be used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "\n",
    "def create_folder_and_ml_table(csv_file, output, delimiter=\",\", encoding=\"ascii\"):\n",
    "    os.makedirs(output, exist_ok=True)\n",
    "    fname = os.path.split(csv_file)[-1]\n",
    "\n",
    "    mltable = {\n",
    "        \"paths\": [{\"file\": f\"./{fname}\"}],\n",
    "        \"transformations\": [\n",
    "            {\"read_delimited\": {\"delimiter\": delimiter, \"encoding\": encoding}}\n",
    "        ],\n",
    "    }\n",
    "    with open(os.path.join(output, \"MLTable\"), \"w\") as f:\n",
    "        f.write(yaml.dump(mltable))\n",
    "    shutil.copy(csv_file, os.path.join(output, fname))\n",
    "\n",
    "\n",
    "train_mltable_path = \"./data/training-mltable-folder\"\n",
    "create_folder_and_ml_table(train_data_path, train_mltable_path)\n",
    "\n",
    "# Training MLTable defined locally, with local data to be uploaded\n",
    "my_training_data_input = Input(type=AssetTypes.MLTABLE, path=train_mltable_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we upload the directory with the test set data which will be used in the batch end point inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(\"test_dataset\", exist_ok=True)\n",
    "shutil.copy(\n",
    "    test_data_path,\n",
    "    \"test_dataset/dominicks_OJ_test.csv\",\n",
    ")\n",
    "\n",
    "my_test_data_input = Input(\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    path=\"test_dataset/\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For documentation on creating your own MLTable assets for jobs beyond this notebook:\n",
    "- https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-mltable details how to write MLTable YAMLs (required for each MLTable asset).\n",
    "- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-data-assets?tabs=Python-SDK covers how to work with them in the v2 CLI/SDK."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Create or Attach existing AmlCompute.\n",
    "[Azure Machine Learning Compute](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) is a managed-compute infrastructure that allows the user to easily create a single or multi-node compute. In this tutorial, you create AmlCompute as your training compute resource.\n",
    "\n",
    "#### Creation of AmlCompute takes approximately 5 minutes.\n",
    "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.core.exceptions import ResourceNotFoundError\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "cluster_name = \"oj-cluster\"\n",
    "\n",
    "try:\n",
    "    # Retrieve an already attached Azure Machine Learning Compute.\n",
    "    compute = ml_client.compute.get(cluster_name)\n",
    "except ResourceNotFoundError as e:\n",
    "    compute = AmlCompute(\n",
    "        name=cluster_name,\n",
    "        size=\"STANDARD_DS12_V2\",\n",
    "        type=\"amlcompute\",\n",
    "        min_instances=0,\n",
    "        max_instances=4,\n",
    "        idle_time_before_scale_down=120,\n",
    "    )\n",
    "    poller = ml_client.begin_create_or_update(compute)\n",
    "    poller.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Configure and run the AutoML Forecasting training job\n",
    "In this section we will configure and run the AutoML job, for training the model.\n",
    "\n",
    "## 4.1 Configure the job through the forecasting() factory function\n",
    "\n",
    "### forecasting() function parameters:\n",
    "\n",
    "The `forecasting()` factory function allows user to configure AutoML for the forecasting task for the most common scenarios with the following properties.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**target_column_name**|The name of the label column.|\n",
    "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
    "|**training_data**|The training data to be used within the experiment. You can use a registered MLTable in the workspace using the format '&lt;mltable_name&gt;:&lt;version&#47;&gt;' OR you can use a local file or folder as a MLTable. For e.g Input(mltable='my_mltable:1') OR Input(mltable=MLTable(local_path=\"./data\")) The parameter 'training_data' must always be provided.|\n",
    "|**compute**|The compute on which the AutoML job will run. In this example we are using a compute called 'oj-cluster' present in the workspace. You can replace it any other compute in the workspace.|\n",
    "|**n_cross_validations**|Number of cross-validation folds to use for model/pipeline selection. This can be set to \"auto\", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or users could specify an integer value.|\n",
    "|**name**|The name of the Job/Run. This is an optional property. If not specified, a random name will be generated.\n",
    "|**experiment_name**|The name of the Experiment. An Experiment is like a folder with multiple runs in Azure ML Workspace that should be related to the same logical machine learning experiment.|\n",
    "|**enable_model_explainability**|If set to true, the explanations for the best model will be generated.|\n",
    "\n",
    "### set_limits() parameters:\n",
    "This is an optional configuration method to configure limits parameters such as timeouts.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**timeout_minutes**|Maximum amount of time in minutes that the whole AutoML job can take before the job terminates. This timeout includes setup, featurization and training runs but does not include the ensembling and model explainability runs at the end of the process since those actions need to happen once all the trials (children jobs) are done. If not specified, the default job's total timeout is 6 days (8,640 minutes). To specify a timeout less than or equal to 1 hour (60 minutes), make sure your dataset's size is not greater than 10,000,000 (rows times column) or an error results.|\n",
    "|**trial_timeout_minutes**|Maximum time in minutes that each trial (child job) can run for before it terminates. If not specified, a value of 1 month or 43200 minutes is used.|\n",
    "|**max_trials**|The maximum number of trials/runs each with a different combination of algorithm and hyperparameters to try during an AutoML job. If not specified, the default is 1000 trials. If using 'enable_early_termination' the number of trials used can be smaller.|\n",
    "|**max_concurrent_trials**|Represents the maximum number of trials (children jobs) that would be executed in parallel. It's a good practice to match this number with the number of nodes your cluster.|\n",
    "|**enable_early_termination**|Whether to enable early termination if the score is not improving in the short term.|\n",
    "\n",
    "### Specialized Forecasting Parameters\n",
    "To define forecasting parameters for your experiment training, you can leverage the .set_forecast_settings() method.\n",
    "The table below details the forecasting parameters we will be passing into our experiment.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**time_column_name**|The name of your time column.|\n",
    "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
    "|**time_series_id_column_names**|The column names used to uniquely identify the time series in data that has multiple rows with the same timestamp. If the time series identifiers are not defined, the data set is assumed to be one time series.|\n",
    "|**frequency**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
    "|**cv_step_size**|Number of periods between two consecutive cross-validation folds. The default value is `None`, in which case AutoMl determines the cross-validation step size automatically. Or users could specify an integer value.|\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# General job parameters.\n",
    "max_trials = 5\n",
    "exp_name = \"dpv2-forecasting-experiment\"\n",
    "target_column_name = \"Quantity\"\n",
    "\n",
    "# Note: we have previously set below parameters.\n",
    "# time_series_id_column_names: [\"Store\", \"Brand\"]\n",
    "# time_column_name: \"WeekStarting\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting the maximum forecast horizon\n",
    "\n",
    "The forecast horizon is the number of periods into the future that the model should predict. It is generally recommend that users set forecast horizons to less than 100 time periods. Furthermore, **AutoML's memory use and computation time increase in proportion to the length of the horizon**, so consider carefully how this value is set. If a long horizon forecast really is necessary, consider aggregating the series to a coarser time scale.\n",
    "\n",
    "Learn more about forecast horizons in our [Auto-train a time-series forecast model](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-auto-train-forecast#configure-and-run-experiment) guide.\n",
    "\n",
    "In this example, we set the forecast horizon to the number of samples per series in the test set (n_test_periods)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_horizon = n_test_periods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customize Featurization Settings\n",
    "\n",
    "The featurization customization in forecasting is an advanced feature in AutoML which allows our customers to change the default forecasting featurization behaviors and column types through `TabularFeaturizationSettings`. The supported scenarios include:\n",
    "\n",
    "1. Column purposes update: Override feature type for the specified column. Currently supports DateTime, Categorical and Numeric. This customization can be used in the scenario that the type of the column cannot correctly reflect its purpose. Some numerical columns, for instance, can be treated as Categorical columns which need to be converted to categorical while some can be treated as epoch timestamp which need to be converted to datetime. To tell our SDK to correctly preprocess these columns, a configuration need to be add with the columns and their desired types.\n",
    "2. Transformer parameters update: Currently supports parameter change for Imputer only. User can customize imputation methods. The supported imputing methods for target column are constant and ffill (forward fill). The supported imputing methods for feature columns are mean, median, most frequent, constant and ffill (forward fill). This customization can be used for the scenario that our customers know which imputation methods fit best to the input data. For instance, some datasets use NaN to represent 0 which the correct behavior should impute all the missing value with 0. To achieve this behavior, these columns need to be configured as constant imputation with `fill_value` 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.automl import ColumnTransformer, TabularFeaturizationSettings\n",
    "\n",
    "# Force the CPWVOL5 feature to be numeric type.\n",
    "column_name_and_types = {\"CPWVOL5\": \"Numeric\"}\n",
    "\n",
    "transformer_params = {\n",
    "    \"Imputer\": [\n",
    "        # Fill missing values in the target column, Quantity, with zeros.\n",
    "        ColumnTransformer(\n",
    "            fields=[\"Quantity\"], parameters={\"strategy\": \"constant\", \"fill_value\": 0}\n",
    "        ),\n",
    "        # Fill missing values in the INCOME column with median value.\n",
    "        ColumnTransformer(fields=[\"INCOME\"], parameters={\"strategy\": \"most_frequent\"}),\n",
    "        # Fill missing values in the Price column with forward fill (last value carried forward).\n",
    "        ColumnTransformer(fields=[\"Price\"], parameters={\"strategy\": \"ffill\"}),\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the AutoML forecasting job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634852262026
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "forecasting-configuration",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "forecasting_job = automl.forecasting(\n",
    "    compute=cluster_name,\n",
    "    experiment_name=exp_name,\n",
    "    training_data=my_training_data_input,\n",
    "    target_column_name=target_column_name,\n",
    "    primary_metric=\"NormalizedRootMeanSquaredError\",\n",
    "    n_cross_validations=\"auto\",\n",
    "    enable_model_explainability=True,\n",
    ")\n",
    "\n",
    "# Limits are all optional\n",
    "forecasting_job.set_limits(\n",
    "    timeout_minutes=60,\n",
    "    trial_timeout_minutes=20,\n",
    "    max_trials=max_trials,\n",
    "    enable_early_termination=True,\n",
    ")\n",
    "\n",
    "# Specify the above custom featurization\n",
    "forecasting_job.set_featurization(\n",
    "    mode=\"custom\",\n",
    "    column_name_and_types=column_name_and_types,\n",
    "    transformer_params=transformer_params,\n",
    ")\n",
    "\n",
    "# Specialized properties for Time Series Forecasting training\n",
    "forecasting_job.set_forecast_settings(\n",
    "    time_column_name=time_column_name,\n",
    "    forecast_horizon=forecast_horizon,\n",
    "    time_series_id_column_names=time_series_id_column_names,\n",
    "    frequency=\"W-THU\",  # Set the forecast frequency to be weekly (start on each Thursday)\n",
    ")\n",
    "\n",
    "# Training properties are optional\n",
    "forecasting_job.set_training(blocked_training_algorithms=[\"ExtremeRandomTrees\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Train the AutoML model\n",
    "Using the `MLClient` created earlier, we will now run this Command in the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit the AutoML job\n",
    "returned_job = ml_client.jobs.create_or_update(\n",
    "    forecasting_job\n",
    ")  # submit the job to the backend\n",
    "\n",
    "print(f\"Created job: {returned_job}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until AutoML training runs are finished\n",
    "ml_client.jobs.stream(returned_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Retrieve the Best Trial (Best Model's trial/run)\n",
    "Use the MLFLowClient to access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Trial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Initialize MLFlow Client\n",
    "The models and artifacts that are produced by AutoML can be accessed via the MLFlow interface.\n",
    "Initialize the MLFlow client here, and set the backend as Azure ML, via. the MLFlow Client.\n",
    "\n",
    "*IMPORTANT*, you need to have installed the latest MLFlow packages with:\n",
    "\n",
    "    pip install azureml-mlflow\n",
    "\n",
    "    pip install mlflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtain the tracking URI for MLFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow\n",
    "\n",
    "# Obtain the tracking URL from MLClient\n",
    "MLFLOW_TRACKING_URI = ml_client.workspaces.get(\n",
    "    name=ml_client.workspace_name\n",
    ").mlflow_tracking_uri\n",
    "\n",
    "print(MLFLOW_TRACKING_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the MLFLOW TRACKING URI\n",
    "\n",
    "mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)\n",
    "\n",
    "print(\"\\nCurrent tracking uri: {}\".format(mlflow.get_tracking_uri()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlflow.tracking.client import MlflowClient\n",
    "\n",
    "# Initialize MLFlow client\n",
    "mlflow_client = MlflowClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the AutoML parent Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = returned_job.name\n",
    "\n",
    "# Example if providing an specific Job name/ID\n",
    "# job_name = \"591640e8-0f88-49c5-adaa-39b9b9d75531\"\n",
    "\n",
    "# Get the parent run\n",
    "mlflow_parent_run = mlflow_client.get_run(job_name)\n",
    "\n",
    "print(\"Parent Run: \")\n",
    "print(mlflow_parent_run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print parent run tags. 'automl_best_child_run_id' tag should be there.\n",
    "print(mlflow_parent_run.data.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the AutoML best child run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the best model's child run\n",
    "best_child_run_id = mlflow_parent_run.data.tags[\"automl_best_child_run_id\"]\n",
    "print(\"Found best child run id: \", best_child_run_id)\n",
    "\n",
    "best_run = mlflow_client.get_run(best_child_run_id)\n",
    "\n",
    "print(\"Best child run: \")\n",
    "print(best_run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Get best model run's validation metrics\n",
    "\n",
    "Access the results (such as models, artifacts, metrics) of a previously completed AutoML Run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.DataFrame(best_run.data.metrics, index=[0]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Model evaluation and deployemnt.\n",
    "## 6.1 Download the best model\n",
    "\n",
    "Access the results (such as models, artifacts, metrics) of a previously completed AutoML Run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create local folder\n",
    "local_dir = \"./artifact_downloads\"\n",
    "if not os.path.exists(local_dir):\n",
    "    os.mkdir(local_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download run's artifacts/outputs\n",
    "local_path = mlflow_client.download_artifacts(\n",
    "    best_run.info.run_id, \"outputs\", local_dir\n",
    ")\n",
    "print(\"Artifacts downloaded in: {}\".format(local_path))\n",
    "print(\"Artifacts: {}\".format(os.listdir(local_path)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Featurization\n",
    "We can look at the engineered feature names generated in time-series featurization via the JSON file named 'engineered_feature_names.json' under the run outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open(os.path.join(local_path, \"engineered_feature_names.json\"), \"r\") as f:\n",
    "    records = json.load(f)\n",
    "\n",
    "records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View featurization summary\n",
    "You can also see what featurization steps were performed on different raw features in the user data. For each raw feature in the user data, the following information is displayed:\n",
    "\n",
    "+ Raw feature name\n",
    "+ Number of engineered features formed out of this raw feature\n",
    "+ Type detected\n",
    "+ If feature was dropped\n",
    "+ List of feature transformations for the raw feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Render the JSON as a pandas DataFrame\n",
    "with open(os.path.join(local_path, \"featurization_summary.json\"), \"r\") as f:\n",
    "    records = json.load(f)\n",
    "fs = pd.DataFrame.from_records(records)\n",
    "\n",
    "# View a summary of the featurization\n",
    "fs[\n",
    "    [\n",
    "        \"RawFeatureName\",\n",
    "        \"TypeDetected\",\n",
    "        \"Dropped\",\n",
    "        \"EngineeredFeatureCount\",\n",
    "        \"Transformations\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Forecasting using batch endpoint<a id=\"forecast\"></a>\n",
    "\n",
    "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. We will do batch inferencing on the test dataset which must have the same schema as training dataset.\n",
    "\n",
    "The inference will run on a remote compute. In this example, it will re-use the training compute.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a model endpoint\n",
    "First, we need to register the model, environment and the batch endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "from azure.ai.ml.entities import (\n",
    "    Environment,\n",
    "    BatchEndpoint,\n",
    "    BatchDeployment,\n",
    "    BatchRetrySettings,\n",
    "    Model,\n",
    ")\n",
    "from azure.ai.ml.constants import BatchDeploymentOutputAction\n",
    "\n",
    "model_name = \"orange-juice-sales\"\n",
    "batch_endpoint_name = \"orange-juice-sales\" + datetime.datetime.now().strftime(\n",
    "    \"%m%d%H%M%f\"\n",
    ")\n",
    "\n",
    "model = Model(\n",
    "    path=f\"azureml://jobs/{best_run.info.run_id}/outputs/artifacts/outputs/model.pkl\",\n",
    "    name=model_name,\n",
    "    description=\"Orange juice sales model.\",\n",
    ")\n",
    "registered_model = ml_client.models.create_or_update(model)\n",
    "\n",
    "env = Environment(\n",
    "    name=\"automl-tabular-env\",\n",
    "    description=\"environment for automl inference\",\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest\",\n",
    "    conda_file=\"artifact_downloads/outputs/conda_env_v_1_0_0.yml\",\n",
    ")\n",
    "\n",
    "endpoint = BatchEndpoint(\n",
    "    name=batch_endpoint_name,\n",
    "    description=\"this is a sample batch endpoint\",\n",
    ")\n",
    "ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a batch deployment, we will use the forecasting_script.py which will load the model and will call the forecast method each time we will envoke the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file = \"forecast.csv\"\n",
    "batch_deployment = BatchDeployment(\n",
    "    name=\"oj-non-mlflow-deployment\",\n",
    "    description=\"this is a sample non-mlflow deployment\",\n",
    "    endpoint_name=batch_endpoint_name,\n",
    "    model=registered_model,\n",
    "    code_path=\"./forecast\",\n",
    "    scoring_script=\"forecasting_script.py\",\n",
    "    environment=env,\n",
    "    environment_variables={\n",
    "        \"TARGET_COLUMN_NAME\": target_column_name,\n",
    "    },\n",
    "    compute=cluster_name,\n",
    "    instance_count=2,\n",
    "    max_concurrency_per_instance=2,\n",
    "    mini_batch_size=10,\n",
    "    output_action=BatchDeploymentOutputAction.APPEND_ROW,\n",
    "    output_file_name=output_file,\n",
    "    retry_settings=BatchRetrySettings(max_retries=3, timeout=30),\n",
    "    logging_level=\"info\",\n",
    "    properties={\"include_output_header\": \"true\"},\n",
    "    tags={\"include_output_header\": \"true\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, start a model deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.begin_create_or_update(batch_deployment).wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to create the Input, representing URI folder, because the batch endpoint is intended to process multiple files at a time. In this example we will use only one test file, which we have uploaded to the blob storage earlier. This file must be available through the url link."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an inference job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = ml_client.batch_endpoints.invoke(\n",
    "    endpoint_name=batch_endpoint_name,\n",
    "    input=my_test_data_input,\n",
    "    deployment_name=\"oj-non-mlflow-deployment\",  # name is required as default deployment is not set\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will stream the job output to monitor the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = job.name\n",
    "batch_job = ml_client.jobs.get(name=job_name)\n",
    "print(batch_job.status)\n",
    "# stream the job logs\n",
    "ml_client.jobs.stream(name=job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the prediction result for metrics calculation\n",
    "The output of forecast output is saved in CSV format. You can use it to calculate test set metrics and plot predictions and actuals over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.jobs.download(job_name, download_path=\".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fcst_df = pd.read_csv(output_file, parse_dates=[time_column_name])\n",
    "fcst_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from metrics_helper import calculate_metrics\n",
    "\n",
    "calculate_metrics(fcst_df[target_column_name], fcst_df[\"predicted\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forecast versus actuals plot.\n",
    "We will join historical data with the predictions to plot predictions and actuals on a time series plot. For illustration purposes, we will select the series for store 2 and \"dominicks\" brand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history_data = pd.read_csv(\n",
    "    \"./data/training-mltable-folder/dominicks_OJ_train.csv\",\n",
    "    parse_dates=[time_column_name],\n",
    ")\n",
    "history_data = history_data.query(\"Store == 2 and Brand == 'dominicks'\")\n",
    "history_data.sort_values(by=time_column_name, inplace=True)\n",
    "history_data = history_data.iloc[-3 * forecast_horizon :]\n",
    "# Merge predictions to historic data.\n",
    "fcst_one = fcst_df.query(\"Store == 2 and Brand == 'dominicks'\")\n",
    "df = pd.concat([history_data, fcst_one], sort=False, ignore_index=True)\n",
    "df.set_index(time_column_name, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.plot(df[[target_column_name, \"predicted\"]])\n",
    "plt.xticks(rotation=45)\n",
    "plt.title(f\"Predicted vs. Actuals\")\n",
    "plt.legend([\"actual\", \"forecast\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the batch endpoint and compute. Do not do it occasionally.\n",
    "ml_client.batch_endpoints.begin_delete(name=batch_endpoint_name).wait()\n",
    "ml_client.compute.begin_delete(name=cluster_name).wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 Deployment\n",
    "After we have tested our model on the batch endpoint, we may want to deploy it as a service. Currently no code deployment using mlflow is not supported for forecasting tasks and we will use the workaround which is described in the Deployment section of the [automl-forecasting-task-energy-demand notebook](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/automl-standalone-jobs/automl-forecasting-task-energy-demand/automl-forecasting-task-energy-demand-advanced-mlflow.ipynb)."
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "jialiu"
   }
  ],
  "category": "tutorial",
  "celltoolbar": "Raw Cell Format",
  "compute": [
   "Remote"
  ],
  "datasets": [
   "Orange Juice Sales"
  ],
  "deployment": [
   "Azure Container Instance"
  ],
  "exclude_from_index": false,
  "framework": [
   "Azure ML AutoML"
  ],
  "friendly_name": "Forecasting orange juice sales with deployment",
  "index_order": 1,
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "tags": [
   "None"
  ],
  "task": "Forecasting",
  "vscode": {
   "interpreter": {
    "hash": "6bd77c88278e012ef31757c15997a7bea8c943977c43d6909403c00ae11d43ca"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}