docs/samples/v1alpha2/custom/kfserving-custom-model/kfserving_sdk_custom_image.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample for KFServing SDK with a custom image\n",
    "\n",
    "This is a sample for KFServing SDK using a custom image.\n",
    "\n",
    "The notebook shows how to use KFServing SDK to create, get and delete InferenceService with a custom image.\n",
    "\n",
    "### Setup\n",
    "- Your `~/.kube/config` should point to a cluster with KFServing installed.\n",
    "- Your cluster's Istio Ingress gateway must be network accessible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the docker image we will be using.\n",
    "\n",
    "The goal of custom image support is to allow users to bring their own wrapped model inside a container and serve it with KFServing. Please note that you will need to ensure that your container is also running a web server e.g. Flask to expose your model endpoints. This example extends kfserving.KFModel which uses the tornado web server.\n",
    "\n",
    "\n",
    "To build and push with Docker Hub set the `DOCKER_HUB_USERNAME` variable below with your Docker Hub username"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this to be your dockerhub username\n",
    "# It will be used when building your image and when creating the InferenceService for your image\n",
    "DOCKER_HUB_USERNAME = \"your_docker_username\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$DOCKER_HUB_USERNAME\"\n",
    "docker build -t $1/kfserving-custom-model ./model-server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$DOCKER_HUB_USERNAME\"\n",
    "docker push $1/kfserving-custom-model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KFServing Client SDK\n",
    "\n",
    "We will use the [KFServing client SDK](https://github.com/kubeflow/kfserving/blob/master/python/kfserving/README.md#kfserving-client) to create the InferenceService and deploy our custom image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kubernetes import client\n",
    "from kubernetes.client import V1Container\n",
    "\n",
    "from kfserving import KFServingClient\n",
    "from kfserving import constants\n",
    "from kfserving import utils\n",
    "from kfserving import V1alpha2EndpointSpec\n",
    "from kfserving import V1alpha2PredictorSpec\n",
    "from kfserving import V1alpha2InferenceServiceSpec\n",
    "from kfserving import V1alpha2InferenceService\n",
    "from kfserving import V1alpha2CustomSpec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "namespace = utils.get_default_target_namespace()\n",
    "print(namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define InferenceService\n",
    "\n",
    "Firstly define default endpoint spec, and then define the inferenceservice using the endpoint spec.\n",
    "\n",
    "To use a custom image we need to use V1alphaCustomSpec which takes a [V1Container](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Container.md)\n",
    " from the kuberenetes library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_version = constants.KFSERVING_GROUP + '/' + constants.KFSERVING_VERSION\n",
    "\n",
    "default_endpoint_spec = V1alpha2EndpointSpec(\n",
    "                          predictor=V1alpha2PredictorSpec(\n",
    "                              custom=V1alpha2CustomSpec(\n",
    "                                  container=V1Container(\n",
    "                                      name=\"kfserving-custom-model\",\n",
    "                                      image=f\"{DOCKER_HUB_USERNAME}/kfserving-custom-model\"))))\n",
    "\n",
    "isvc = V1alpha2InferenceService(api_version=api_version,\n",
    "                          kind=constants.KFSERVING_KIND,\n",
    "                          metadata=client.V1ObjectMeta(\n",
    "                              name='kfserving-custom-model', namespace=namespace),\n",
    "                          spec=V1alpha2InferenceServiceSpec(default=default_endpoint_spec))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the InferenceService\n",
    "\n",
    "Call KFServingClient to create InferenceService."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KFServing = KFServingClient()\n",
    "KFServing.create(isvc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the InferenceService"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KFServing.get('kfserving-custom-model', namespace=namespace, watch=True, timeout_seconds=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run a prediction "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"kfserving-custom-model\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash --out CLUSTER_IP\n",
    "INGRESS_GATEWAY=\"istio-ingressgateway\"\n",
    "echo \"$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$MODEL_NAME\" --out SERVICE_HOSTNAME\n",
    "echo \"$(kubectl get inferenceservice $1 -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "with open('input.json') as json_file:\n",
    "    data = json.load(json_file)\n",
    "    url = f\"http://{CLUSTER_IP.strip()}/v1/models/{MODEL_NAME}:predict\"\n",
    "    headers = {\"Host\": SERVICE_HOSTNAME.strip()}\n",
    "    result = requests.post(url, data=json.dumps(data), headers=headers)\n",
    "    print(result.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the InferenceService"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KFServing.delete(MODEL_NAME, namespace=namespace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}