diff --git a/docs/docs/APIGuide/PipelineAPI/inference.md b/docs/docs/APIGuide/PipelineAPI/inference.md index 0587e46382..a84c47c4ab 100644 --- a/docs/docs/APIGuide/PipelineAPI/inference.md +++ b/docs/docs/APIGuide/PipelineAPI/inference.md @@ -3,35 +3,68 @@ ## Overview Inference is a package in Analytics Zoo aiming to provide high level APIs to speed-up development. It -allows user to conveniently use pre-trained models from Analytics Zoo, Tensorflow and Caffe. -Inference provides multiple Java/Scala interfaces. +allows user to conveniently use pre-trained models from Analytics Zoo, Caffe, Tensorflow and OpenVINO Intermediate Representation(IR). +Inference provides multiple Scala interfaces. ## Highlights -1. Easy-to-use java/scala API for loading and prediction with deep learning models. +1. Easy-to-use APIs for loading and prediction with deep learning models of Analytics Zoo, Caffe, Tensorflow and OpenVINO Intermediate Representation(IR). -2. Support transformation of various input data type, thus supporting future prediction tasks +2. Support transformation of various input data type, thus supporting future prediction tasks. -## Primary APIs for Java +3. Transparently support the OpenVINO toolkit, which deliver a significant boost for inference speed ([up to 19.9x](https://software.intel.com/en-us/blogs/2018/05/15/accelerate-computer-vision-from-edge-to-cloud-with-openvino-toolkit)). +## Primary APIs for Java **load** AbstractInferenceModel provides `load` API for loading a pre-trained model, thus we can conveniently load various kinds of pre-trained models in java applications. The load result of -`AbstractInferenceModel` is a [`FloatInferenceModel`](https://github.com/xuex2017/analytics-zoo/blob/88afc2d921bb50341d8d7e02d380fa28f49d246b/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/inference/FloatInferenceModel.scala). +`AbstractInferenceModel` is an `AbstractModel`. We just need to specify the model path and optionally weight path if exists where we previously saved the model. +***load*** + +`load` method is to load a BigDL model. + +***loadCaffe*** + +`loadCaffe` method is to load a caffe model. + +***loadTF*** + +`loadTF` method is to load a tensorflow model. There are two backends to load a tensorflow model and to do the predictions: TFNet and OpenVINO. For OpenVINO backend, supported tensorflow models are listed below: + + faster_rcnn_inception_resnet_v2_atrous_coco + faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco + faster_rcnn_inception_resnet_v2_atrous_lowproposals_oid + faster_rcnn_inception_resnet_v2_atrous_oid + faster_rcnn_nas_coco + faster_rcnn_nas_lowproposals_coco + faster_rcnn_resnet101_coco + faster_rcnn_resnet101_kitti + faster_rcnn_resnet101_lowproposals_coco + mask_rcnn_inception_resnet_v2_atrous_coco + mask_rcnn_inception_v2_coco + mask_rcnn_resnet101_atrous_coco + mask_rcnn_resnet50_atrous_coco + ssd_inception_v2_coco + ssd_mobilenet_v1_coco + ssd_mobilenet_v2_coco + ssdlite_mobilenet_v2_coco + +***loadOpenVINO*** + +`loadOpenVINO` method is to load an OpenVINO Intermediate Representation(IR). **predict** AbstractInferenceModel provides `predict` API for prediction with loaded model. The predict result of`AbstractInferenceModel` is a `List>` by default. - -**Java example** +## Examples It's very easy to apply abstract inference model for inference with below code piece. You will need to write a subclass that extends AbstractinferenceModel. @@ -51,6 +84,54 @@ List> result = model.predict(inputList); ## Primary APIs for Scala +**InferenceModel** + +`InferenceModel` is a thead-safe wrapper of AbstractModels, which can be used to load models and do the predictions. + +***doLoad*** + +`doLoad` method is to load a bigdl, analytics-zoo model. + +***doLoadCaffe*** + +`doLoadCaffe` method is to load a caffe model. + +***doLoadTF*** + +`doLoadTF` method is to load a tensorflow model. The model can be loaded as a `FloatModel` or an `OpenVINOModel`. There are two backends to load a tensorflow model: TFNet and OpenVINO. + +For OpenVINO backend, supported tensorflow models are listed below: + + faster_rcnn_inception_resnet_v2_atrous_coco + faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco + faster_rcnn_inception_resnet_v2_atrous_lowproposals_oid + faster_rcnn_inception_resnet_v2_atrous_oid + faster_rcnn_nas_coco + faster_rcnn_nas_lowproposals_coco + faster_rcnn_resnet101_coco + faster_rcnn_resnet101_kitti + faster_rcnn_resnet101_lowproposals_coco + mask_rcnn_inception_resnet_v2_atrous_coco + mask_rcnn_inception_v2_coco + mask_rcnn_resnet101_atrous_coco + mask_rcnn_resnet50_atrous_coco + ssd_inception_v2_coco + ssd_mobilenet_v1_coco + ssd_mobilenet_v2_coco + ssdlite_mobilenet_v2_coco + +***doLoadOpenVINO*** + +`doLoadOpenVINO` method is to load an OpenVINO Intermediate Representation(IR). + +***doReload*** + +`doReload` method is to reload the bigdl, analytics-zoo model. + +***doPredict*** + +`doPredict` method is to do the prediction. + **InferenceSupportive** `InferenceSupportive` is a trait containing several methods for type transformation, which transfer a model input @@ -60,17 +141,27 @@ For example, method `transferTensorToJTensor` convert a model input of data type to [`JTensor`](https://github.com/intel-analytics/analytics-zoo/blob/88afc2d921bb50341d8d7e02d380fa28f49d246b/zoo/src/main/java/com/intel/analytics/zoo/pipeline/inference/JTensor.java) , which will be the input for a FloatInferenceModel. -**FloatInferenceModel** +**AbstractModel** + +`AbstractModel` is an abstract class to provide APIs for basic functions - `predict` interface for prediction, `copy` interface for coping the model into the queue of AbstractModels, `release` interface for releasing the model and `isReleased` interface for checking the state of model release. + +**FloatModel** -`FloatInferenceModel` is an extending class of `InferenceSupportive` and additionally provides `predict` API for prediction tasks. +`FloatModel` is an extending class of `AbstractModel` and achieves all `AbstractModel` interfaces. + +**OpenVINOModel** + +`OpenVINOModel` is an extending class of `AbstractModel`. It achieves all `AbstractModel` functions. **InferenceModelFactory** -`InferenceModelFactory` is an object with APIs for loading pre-trained Analytics Zoo models, Caffe models and Tensorflow models. -We just need to specify the model path and optionally weight path if exists where we previously saved the model. -The load result of is a `FloatInferenceModel`. +`InferenceModelFactory` is an object with APIs for loading pre-trained Analytics Zoo models, Caffe models, Tensorflow models and OpenVINO Intermediate Representations(IR). +Analytics Zoo models, Caffe models, Tensorflow models can be loaded as FloatModels. The load result of it is a `FloatModel` +Tensorflow models and OpenVINO Intermediate Representations(IR) can be loaded as OpenVINOModels. The load result of it is an `OpenVINOModel`. +The load result of it is a `FloatModel` or an `OpenVINOModel`. -**ModelLoader** +**OpenVinoInferenceSupportive** -`ModelLoader` is an extending object of `InferenceSupportive` and focus on the implementation of loading pre-trained models \ No newline at end of file +`OpenVinoInferenceSupportive` is an extending object of `InferenceSupportive` and focus on the implementation of loading pre-trained models, including tensorflow models and OpenVINO Intermediate Representations(IR). +There are two backends to load a tensorflow model: TFNet and OpenVINO. For OpenVINO backend, [supported tensorflow models](#jump) are listed in the section of `doLoadTF` method of `InferenceModel` API above. diff --git a/zoo/src/main/java/com/intel/analytics/zoo/pipeline/inference/abstract-Inference-model.md b/zoo/src/main/java/com/intel/analytics/zoo/pipeline/inference/abstract-Inference-model.md index ed82431088..111c348eeb 100644 --- a/zoo/src/main/java/com/intel/analytics/zoo/pipeline/inference/abstract-Inference-model.md +++ b/zoo/src/main/java/com/intel/analytics/zoo/pipeline/inference/abstract-Inference-model.md @@ -15,7 +15,8 @@ develop your java applications. 2. In a few lines, run large scale inference from pre-trained models of Analytics-Zoo, Caffe, Tensorflow and OpenVINO Intermediate Representation(IR). -3. Transparently support the OpenVINO toolkit, which deliver a significant boost for inference speed (up to 19.9x). +3. Transparently support the OpenVINO toolkit, which deliver a significant boost for inference speed ([up to 19.9x](https://software.intel.com/en-us/blogs/2018/05/15/accelerate-computer-vision-from-edge-to-cloud-with-openvino-toolkit)). + ## Primary APIs diff --git a/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/inference/inference.md b/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/inference/inference.md index 5f2717f03e..15566700c6 100644 --- a/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/inference/inference.md +++ b/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/inference/inference.md @@ -14,7 +14,7 @@ Inference provides multiple Scala interfaces. 2. Support transformation of various input data type, thus supporting future prediction tasks. -3. Transparently support the OpenVINO toolkit, which deliver a significant boost for inference speed (up to 19.9x). +3. Transparently support the OpenVINO toolkit, which deliver a significant boost for inference speed ([up to 19.9x](https://software.intel.com/en-us/blogs/2018/05/15/accelerate-computer-vision-from-edge-to-cloud-with-openvino-toolkit)). ## Primary APIs