deeplens-worksite-safety-public.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Worksite Safety - Hardhat detection\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "2. [Prerequisites and Preprocessing](#Prequisites-and-Preprocessing)\n",
    "3. [Fine-tuning the Image classification model](#Fine-tuning-the-Image-classification-model)\n",
    "4. [Set up hosting for the model](#Set-up-hosting-for-the-model)\n",
    "  1. [Import model into hosting](#Import-model-into-hosting)\n",
    "  2. [Create endpoint configuration](#Create-endpoint-configuration)\n",
    "  3. [Create endpoint](#Create-endpoint)\n",
    "5. [Perform Inference](#Perform-Inference)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Introduction\n",
    "\n",
    "Pre-requisite - We want a model that can inspect an image of a person and infer whether they are wearing a hardhat or not, so we need to train the model with lots of images or individuals wearing hardhats, wearing no hats, and wearing hats that are not hardhats. I have taken photos using a standard mobile phone of colleagues in various locations, orientations and lighting conditions for each of the above scenarios. I have about 1400 photos for compliant, 1100 photos for not_compliant, 200 photos for unknown and 100 photos for unsure. This is a bit light on, a real training exercise would want to extend this, perhaps as many as 5000 photos for each scenario. To improve the training results we will do some simple augmentation to create more training images from our base set. In this case we we will create a vertically flipped copy of every image so as to double our training set.\n",
    "\n",
    "We will then use the Amazon sagemaker image classification algorithm in transfer learning mode to fine-tune a pre-trained model (trained on imagenet data) to learn to classify this dataset for hardhat detection\n",
    "\n",
    "To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prequisites and Preprocessing\n",
    "\n",
    "### Permissions and environment variables\n",
    "\n",
    "Here we set up the linkage and authentication to AWS services. There are three parts to this:\n",
    "\n",
    "* The roles used to give learning and hosting access to your data. This will automatically be obtained from the role used to start the notebook\n",
    "* The S3 bucket that you want to use for training and model data\n",
    "* The Amazon sagemaker image classification docker image which need not be changed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import boto3\n",
    "import re\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "bucket_path=\"your-bucket-name-and-path\" # customize to your bucket name and optionally path\n",
    "\n",
    "training_image = get_image_uri(boto3.Session().region_name, 'image-classification')\n",
    "\n",
    "print(training_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensure all images are in S3 bucket (in preferred format)\n",
    "\n",
    "The SageMaker Image Classification algorithm uses mxnet and prefers input in recordio format. I stored my images in the following folder structure on my local machine to facilitate conversion of the image files to recordio format\n",
    "In this example I use an image width and length of 224 pixels\n",
    "\n",
    "    images\n",
    "        /0_compliant\n",
    "        /1_notcompliant\n",
    "        /2_unsure\n",
    "        /3_nosubject\n",
    "   \n",
    "\n",
    "### For images of size 224\n",
    "python im2rec.py --list --recursive --train-ratio 0.95 dataset images_all\n",
    "\n",
    "python im2rec.py --resize 224 dataset images\n",
    "\n",
    "aws s3 cp dataset_train.rec s3://your-bucket-name-and-path/train/\n",
    "aws s3 cp dataset_224_val.rec s3://your-bucket-name-and-path/validation/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning the Image classification model\n",
    "\n",
    "The imageset we are using consists of 5156 training images and 272 validation images over 4 categories \n",
    "    0_compliant\n",
    "    1_notcompliant\n",
    "    2_unsure\n",
    "    3_nosubject\n",
    "\n",
    "\n",
    "Once we have the data available in the correct format for training, the next step is to actually train the model using the data. Before training the model, we need to setup the training parameters. The next section will explain the parameters in detail.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training parameters\n",
    "There are two kinds of parameters that need to be set for training. The first one are the parameters for the training job. These include:\n",
    "\n",
    "* **Input specification**: These are the training and validation channels that specify the path where training data is present. These are specified in the \"InputDataConfig\" section. The main parameters that need to be set is the \"ContentType\" which can be set to \"application/x-recordio\" or \"application/x-image\" based on the input data format and the S3Uri which specifies the bucket and the folder where the data is present. \n",
    "* **Output specification**: This is specified in the \"OutputDataConfig\" section. We just need to specify the path where the output can be stored after training\n",
    "* **Resource config**: This section specifies the type of instance on which to run the training and the number of hosts used for training. If \"InstanceCount\" is more than 1, then training can be run in a distributed manner. \n",
    "\n",
    "Apart from the above set of parameters, there are hyperparameters that are specific to the algorithm. These are:\n",
    "\n",
    "* **num_layers**: The number of layers (depth) for the network. We use 18 in this samples but other values such as 50, 152 can be used.\n",
    "* **num_training_samples**: This is the total number of training samples. It is set to 1875 for the input dataset with the current split\n",
    "* **num_classes**: This is the number of output classes for the new dataset. Imagenet was trained with 1000 output classes but the number of output classes can be changed for fine-tuning. For worksite safety, we use 2 because it has 2 categories: 0_yeshardhat and 1_nohardhat\n",
    "* **epochs**: Number of training epochs\n",
    "* **learning_rate**: Learning rate for training\n",
    "* **mini_batch_size**: The number of training samples used for each mini batch. In distributed training, the number of training samples used per batch will be N * mini_batch_size where N is the number of hosts on which training is run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "# The algorithm supports multiple network depth (number of layers). They are 18, 34, 50, 101, 152 and 200\n",
    "# For this training, we will use 101 layers\n",
    "num_layers = 50\n",
    "# we need to specify the input image shape for the training data\n",
    "image_shape = \"3,224,224\"\n",
    "# we also need to specify the number of training samples in the training set\n",
    "# for our dataset this is 5308.\n",
    "# An easy way to get this is to wc -l dataset_train.lst\n",
    "num_training_samples = 5308\n",
    "# specify the number of output classes\n",
    "num_classes = 4\n",
    "# batch size for training\n",
    "mini_batch_size =  128\n",
    "# number of epochs\n",
    "epochs = 2\n",
    "# learning rate\n",
    "learning_rate = 0.01\n",
    "top_k=2\n",
    "# Since we are using transfer learning, we set use_pretrained_model to 1 so that weights can be \n",
    "# initialized with pre-trained weights\n",
    "use_pretrained_model = 1\n",
    "augmentation_type = \"crop_color_transform\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import time\n",
    "import boto3\n",
    "from time import gmtime, strftime\n",
    "\n",
    "\n",
    "s3 = boto3.client('s3')\n",
    "# create unique job name \n",
    "job_name_prefix = \"your-jobname-prefix\"\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "job_name = job_name_prefix + timestamp\n",
    "training_params = \\\n",
    "{\n",
    "    # specify the training docker image\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": training_image,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": 's3://{}/{}/output'.format(bucket, job_name_prefix)\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.p3.16xlarge\",\n",
    "        \"VolumeSizeInGB\": 50\n",
    "    },\n",
    "    \"TrainingJobName\": job_name,\n",
    "    \"HyperParameters\": {\n",
    "        \"image_shape\": image_shape,\n",
    "        \"num_layers\": str(num_layers),\n",
    "        \"num_training_samples\": str(num_training_samples),\n",
    "        \"num_classes\": str(num_classes),\n",
    "        \"mini_batch_size\": str(mini_batch_size),\n",
    "        \"epochs\": str(epochs),\n",
    "        \"learning_rate\": str(learning_rate),\n",
    "        \"use_pretrained_model\": str(use_pretrained_model),\n",
    "        \"augmentation_type\": augmentation_type\n",
    "    },\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 360000\n",
    "    },\n",
    "#Training data should be inside a subdirectory called \"train\"\n",
    "#Validation data should be inside a subdirectory called \"validation\"\n",
    "#The algorithm currently only supports fullyreplicated model (where data is copied onto each machine)\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": 's3://{}/train/'.format(bucket),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/x-recordio\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": 's3://{}/validation/'.format(bucket),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/x-recordio\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "print('Training job name: {}'.format(job_name))\n",
    "print('\\nInput Data Location: {}'.format(training_params['InputDataConfig'][0]['DataSource']['S3DataSource']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After setting training parameters, we kick off training, and poll for status until training is completed, which in this example, takes around 4 minutes per epoch on a p3.16xlarge machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# create the Amazon SageMaker training job\n",
    "sagemaker = boto3.client(service_name='sagemaker')\n",
    "sagemaker.create_training_job(**training_params)\n",
    "\n",
    "# confirm that the training job has started\n",
    "status = sagemaker.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']\n",
    "print('Training job current status: {}'.format(status))\n",
    "\n",
    "try:\n",
    "    # wait for the job to finish and report the ending status\n",
    "    sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=job_name)\n",
    "    training_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "    status = training_info['TrainingJobStatus']\n",
    "    print(\"Training job ended with status: \" + status)\n",
    "except:\n",
    "    print('Training failed to start')\n",
    "     # if exception is raised, that means it has failed\n",
    "    message = sagemaker.describe_training_job(TrainingJobName=job_name)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you see the message,\n",
    "\n",
    "> `Training job ended with status: Completed`\n",
    "\n",
    "then that means training sucessfully completed and the output model was stored in the output path specified by `training_params['OutputDataConfig']`.\n",
    "\n",
    "You can also view information about and the status of a training job using the AWS SageMaker console. Just click on the \"Jobs\" tab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "***\n",
    "\n",
    "A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the topic mixture representing a given document.\n",
    "\n",
    "This section involves several steps,\n",
    "\n",
    "1. [Create Model](#CreateModel) - Create model for the training output\n",
    "1. [Create Endpoint Configuration](#CreateEndpointConfiguration) - Create a configuration defining an endpoint.\n",
    "1. [Create Endpoint](#CreateEndpoint) - Use the configuration to create an inference endpoint.\n",
    "1. [Perform Inference](#Perform Inference) - Perform inference on some input data using the endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Model\n",
    "\n",
    "We now create a SageMaker Model from the training output. Using the model we can create an Endpoint Configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import boto3\n",
    "from time import gmtime, strftime\n",
    "\n",
    "sage = boto3.Session().client(service_name='sagemaker') \n",
    "\n",
    "model_name=\"your-modelname\"\n",
    "print(model_name)\n",
    "info = sage.describe_training_job(TrainingJobName=job_name)\n",
    "model_data = info['ModelArtifacts']['S3ModelArtifacts']\n",
    "print(model_data)\n",
    "\n",
    "hosting_image = get_image_uri(boto3.Session().region_name, 'image-classification')\n",
    "\n",
    "primary_container = {\n",
    "    'Image': hosting_image,\n",
    "    'ModelDataUrl': model_data,\n",
    "}\n",
    "\n",
    "create_model_response = sage.create_model(\n",
    "    ModelName = model_name,\n",
    "    ExecutionRoleArn = role,\n",
    "    PrimaryContainer = primary_container)\n",
    "\n",
    "print(create_model_response['ModelArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Endpoint Configuration\n",
    "Sagemaker supports configuring REST endpoints hosting multiple models, e.g. for A/B testing purposes. In order to support this, customers create an endpoint configuration, that describes the distribution of traffic across the models, whether split, shadowed, or sampled in some way.\n",
    "\n",
    "In addition, the endpoint configuration describes the instance type required for model deployment, and at launch will describe the autoscaling configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime\n",
    "\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "endpoint_config_name = job_name_prefix + '-epc-' + timestamp\n",
    "endpoint_config_response = sage.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType':'ml.t2.medium',\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName':model_name,\n",
    "        'VariantName':'AllTraffic'}])\n",
    "\n",
    "print('Endpoint configuration name: {}'.format(endpoint_config_name))\n",
    "print('Endpoint configuration arn:  {}'.format(endpoint_config_response['EndpointConfigArn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Endpoint\n",
    "Lastly, the customer creates the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import time\n",
    "\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "endpoint_name = job_name_prefix + '-ep-' + timestamp\n",
    "print('Endpoint name: {}'.format(endpoint_name))\n",
    "\n",
    "endpoint_params = {\n",
    "    'EndpointName': endpoint_name,\n",
    "    'EndpointConfigName': endpoint_config_name,\n",
    "}\n",
    "endpoint_response = sagemaker.create_endpoint(**endpoint_params)\n",
    "print('EndpointArn = {}'.format(endpoint_response['EndpointArn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for endpoint creation to complete\n",
    "Let's check the status of the endpoint and wait for creation to complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the status of the endpoint\n",
    "response = sagemaker.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = response['EndpointStatus']\n",
    "print('EndpointStatus = {}'.format(status))\n",
    "\n",
    "\n",
    "# wait until the status has changed\n",
    "sagemaker.get_waiter('endpoint_in_service').wait(EndpointName=endpoint_name)\n",
    "\n",
    "\n",
    "# print the status of the endpoint\n",
    "endpoint_response = sagemaker.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = endpoint_response['EndpointStatus']\n",
    "print('Endpoint creation ended with EndpointStatus = {}'.format(status))\n",
    "\n",
    "if status != 'InService':\n",
    "    raise Exception('Endpoint creation failed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you see the message,\n",
    "\n",
    "> `Endpoint creation ended with EndpointStatus = InService`\n",
    "\n",
    "then congratulations! You now have a functioning inference endpoint. You can confirm the endpoint configuration and status by navigating to the \"Endpoints\" tab in the AWS SageMaker console.\n",
    "\n",
    "We will finally create a runtime object from which we can invoke the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "runtime = boto3.Session().client(service_name='runtime.sagemaker') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download test images\n",
    "This is where you will have a bunch of jpeg images for testing. \n",
    "Make sure the images are of the correct dimensions (max 224x224 pixels for this model)\n",
    "\n",
    "Now let's display and run inference on a few of our test images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!aws s3 cp --recursive s3://deeplens-your-bucketpath-name/test /tmp/test #Change the name of the bucketpath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate Model\n",
    "We will now validate the model for use. Obtain the endpoint from the client library using the result from previous operations, and generate classifications from the trained model using that endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "from IPython.display import Image\n",
    "\n",
    "file_name = '/tmp/test/sample_image1.jpg'\n",
    "display(Image(file_name,width=200))\n",
    "\n",
    "with open(file_name, 'rb') as f:\n",
    "    payload = f.read()\n",
    "    payload = bytearray(payload)\n",
    "response = runtime.invoke_endpoint(EndpointName=endpoint_name, \n",
    "                                   ContentType='application/x-image', \n",
    "                                   Body=payload)\n",
    "result = response['Body'].read()\n",
    "# result will be in json format and convert it to ndarray\n",
    "result = json.loads(result)\n",
    "# the result will output the probabilities for all classes\n",
    "# find the class with maximum probability and print the class index\n",
    "index = np.argmax(result)\n",
    "inference_categories = ['compliant', 'not-compliant', 'unknown', 'nosubject']\n",
    "print(\"Result: label - \" + inference_categories[index] + \", probability - \" + str(result[index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up\n",
    "\n",
    "When we're done with the endpoint, we can just delete it and the backing instances will be released.  Run the following cell to delete the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage.delete_endpoint(EndpointName=endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_mxnet_p36",
   "language": "python",
   "name": "conda_mxnet_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}