From 8bb350c33ea20bf75c3450bfa8752eb33bd47a72 Mon Sep 17 00:00:00 2001 From: Tom Allsop Date: Thu, 26 Aug 2021 14:20:40 +0100 Subject: [PATCH] Added accuracy scores to table --- README.md | 166 +++++++++++------- .../micronet_medium/tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../micronet_small/tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../tflite_uint8/README.md | 2 +- .../cnn_large/tflite_int8/README.md | 3 +- .../cnn_large/tflite_int8/definition.yaml | 2 +- .../cnn_medium/tflite_int8/README.md | 2 +- .../cnn_medium/tflite_int8/definition.yaml | 2 +- .../cnn_small/tflite_int8/README.md | 4 +- .../cnn_small/tflite_int8/definition.yaml | 2 +- .../dnn_large/tflite_int8/README.md | 2 +- .../dnn_large/tflite_int8/definition.yaml | 2 +- .../dnn_medium/tflite_int8/README.md | 2 +- .../dnn_medium/tflite_int8/definition.yaml | 2 +- .../dnn_small/tflite_int8/README.md | 2 +- .../dnn_small/tflite_int8/definition.yaml | 2 +- .../tflite_clustered_fp32/README.md | 2 +- .../tflite_clustered_fp32/definition.yaml | 2 +- .../tflite_clustered_int8/README.md | 4 +- .../tflite_clustered_int8/definition.yaml | 2 +- .../ds_cnn_large/tflite_int8/README.md | 4 +- .../ds_cnn_large/tflite_int8/definition.yaml | 2 +- .../ds_cnn_medium/tflite_int8/README.md | 2 +- .../ds_cnn_medium/tflite_int8/definition.yaml | 2 +- .../ds_cnn_small/tflite_int8/README.md | 2 +- .../ds_cnn_small/tflite_int8/definition.yaml | 2 +- .../micronet_large/tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../micronet_medium/tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../micronet_small/tflite_int8/README.md | 2 +- .../tflite_int8/definition.yaml | 2 +- .../ssd_mobilenet_v1/tflite_fp32/README.md | 2 +- .../tflite_fp32/definition.yaml | 2 +- .../tflite_int8/definition.yaml | 2 +- .../ssd_mobilenet_v1/tflite_uint8/README.md | 4 +- .../tflite_uint8/definition.yaml | 2 +- .../wav2letter/tflite_int8/README.md | 2 +- .../wav2letter/tflite_int8/definition.yaml | 2 +- .../wav2letter/tflite_pruned_int8/README.md | 2 +- .../tflite_pruned_int8/definition.yaml | 2 +- .../micronet_vww2/tflite_int8/README.md | 2 +- .../micronet_vww2/tflite_int8/definition.yaml | 2 +- .../micronet_vww3/tflite_int8/README.md | 2 +- .../micronet_vww3/tflite_int8/definition.yaml | 2 +- .../micronet_vww4/tflite_int8/README.md | 2 +- .../micronet_vww4/tflite_int8/definition.yaml | 2 +- 51 files changed, 159 insertions(+), 116 deletions(-) diff --git a/README.md b/README.md index e56011c..ce3f069 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ -# Model Zoo +# Model Zoo ![version](https://img.shields.io/badge/version-21.05-0091BD) > A collection of machine learning models optimized for Arm IP. + ## Anomaly Detection @@ -9,40 +10,46 @@ - - - - + + + + + - + + - + + - + +
Network Type FrameworkCortex-ACortex-MMali GPUEthos UCortex-ACortex-MMali GPUEthos UScore (AUC)
MicroNet Large INT8MicroNet Large INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.968
MicroNet Medium INT8MicroNet Medium INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.963
MicroNet Small INT8MicroNet Small INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.955
+**Dataset**: Dcase 2020 Task 2 Slide Rail + ## Image Classification @@ -50,10 +57,11 @@ - - - - + + + + + @@ -63,18 +71,22 @@ + - + +
Network Type FrameworkCortex-ACortex-MMali GPUEthos UCortex-ACortex-MMali GPUEthos UScore (Top 1 Accuracy)
MobileNet v2 1.0 224 INT8 *:heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.697
MobileNet v2 1.0 224 UINT8 UINT8 TensorFlow Lite:heavy_check_mark::heavy_multiplication_x: :heavy_multiplication_x: :heavy_check_mark: :heavy_check_mark:0.708
+**Dataset**: ILSVRC 2012 + ## Keyword Spotting @@ -82,91 +94,101 @@ - - - - + + + + + - + + - - + + - - + + + - + + - + + - + + - + + - - + + + - - + + - + + - + + - + @@ -176,6 +198,7 @@ + @@ -185,36 +208,42 @@ + - + + - + + - + +
Network Type FrameworkCortex-ACortex-MMali GPUEthos UCortex-ACortex-MMali GPUEthos UScore (Accuracy)
DS-CNN Clustered INT8 *CNN Large INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.929
DS-CNN Clustered FP32 *FP32CNN Medium INT8 *INT8 TensorFlow Lite :heavy_check_mark::heavy_multiplication_x: :heavy_check_mark::heavy_multiplication_x::heavy_check_mark::heavy_check_mark:0.913
CNN Large INT8 *CNN Small INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.914
CNN Medium INT8 *DNN Large INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.863
CNN Small INT8 *DNN Medium INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.846
DNN Large INT8 *DNN Small INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.827
DNN Medium INT8 *INT8DS-CNN Clustered FP32 *FP32 TensorFlow Lite :heavy_check_mark::heavy_multiplication_x: :heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_multiplication_x:0.950
DNN Small INT8 *DS-CNN Clustered INT8 * INT8 TensorFlow Lite:heavy_multiplication_x: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark::heavy_check_mark:0.940
DS-CNN Large INT8 * INT8 TensorFlow Lite:heavy_multiplication_x: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark::heavy_check_mark:0.946
DS-CNN Medium INT8 *:heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.934
DS-CNN Small INT8 *:heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.934
MicroNet Large INT8MicroNet Large INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.965
MicroNet Medium INT8MicroNet Medium INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.958
MicroNet Small INT8MicroNet Small INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark:0.953
+**Dataset**: Google Speech Commands Test Set + ## Object Detection @@ -222,37 +251,41 @@ - - - - + + + + + - - + + + - - + + + - + + @@ -262,9 +295,12 @@ +
Network Type FrameworkCortex-ACortex-MMali GPUEthos UCortex-ACortex-MMali GPUEthos UScore (mAP)
SSD MobileNet v1 INT8 *INT8SSD MobileNet v1 FP32 *FP32 TensorFlow Lite :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x:0.210
SSD MobileNet v1 FP32 *FP32SSD MobileNet v1 INT8 *UINT8 TensorFlow Lite :heavy_check_mark: :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x:0.234
SSD MobileNet v1 UINT8 * UINT8 TensorFlow Lite:heavy_check_mark::heavy_multiplication_x: :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x:0.180
YOLO v3 Tiny FP32 *:heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x:0.331
+**Dataset**: COCO Validation 2017 + ## Speech Recognition @@ -272,31 +308,35 @@ - - - - + + + + + - + + - + +
Network Type FrameworkCortex-ACortex-MMali GPUEthos UCortex-ACortex-MMali GPUEthos UScore (LER)
Wav2letter Pruned INT8 *Wav2letter INT8 INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.0877
Wav2letter INT8Wav2letter Pruned INT8 * INT8 TensorFlow Lite :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:0.0783
+**Dataset**: LibriSpeech ## Visual Wake Words @@ -305,45 +345,51 @@ Network Type Framework - Cortex-A - Cortex-M - Mali GPU - Ethos U + Cortex-A + Cortex-M + Mali GPU + Ethos U + Score (Accuracy) - MicroNet VWW-4 INT8 + MicroNet VWW-2 INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_multiplication_x: + 0.768 - MicroNet VWW-3 INT8 + MicroNet VWW-3 INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_multiplication_x: + 0.855 - MicroNet VWW-2 INT8 + MicroNet VWW-4 INT8 INT8 TensorFlow Lite :heavy_multiplication_x: :heavy_check_mark: :heavy_multiplication_x: :heavy_multiplication_x: + 0.822 +**Dataset**: Visual Wake Words + + ### Key * :heavy_check_mark: - Will run on this platform. * :heavy_multiplication_x: - Will not run on this platform. * `*` - Code to recreate model available. - ## License -[Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) unless otherwise explicitly stated. \ No newline at end of file +[Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) unless otherwise explicitly stated. diff --git a/models/anomaly_detection/micronet_medium/tflite_int8/README.md b/models/anomaly_detection/micronet_medium/tflite_int8/README.md index 00bb3bf..5cb9f64 100644 --- a/models/anomaly_detection/micronet_medium/tflite_int8/README.md +++ b/models/anomaly_detection/micronet_medium/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Dcase 2020 Task 2 Slide Rail | Metric | Value | |--------|-------| -| AUC | 0.9632 | +| AUC | 0.963 | ## Optimizations | Optimization | Value | diff --git a/models/anomaly_detection/micronet_medium/tflite_int8/definition.yaml b/models/anomaly_detection/micronet_medium/tflite_int8/definition.yaml index 019d583..3c1c81e 100644 --- a/models/anomaly_detection/micronet_medium/tflite_int8/definition.yaml +++ b/models/anomaly_detection/micronet_medium/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: DCASE 2020 Task 2 Slide rail: - AUC: 0.9632 + AUC: 0.963 description: This is a fully quantized version (asymmetrical int8) of the MicroNet Medium model developed by Arm, from the MicroNets paper. It is trained on the 'slide rail' task from http://dcase.community/challenge2020/task-unsupervised-detection-of-anomalous-sounds. diff --git a/models/anomaly_detection/micronet_small/tflite_int8/README.md b/models/anomaly_detection/micronet_small/tflite_int8/README.md index 8e8386c..7bc91ab 100644 --- a/models/anomaly_detection/micronet_small/tflite_int8/README.md +++ b/models/anomaly_detection/micronet_small/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Dcase 2020 Task 2 Slide Rail | Metric | Value | |--------|-------| -| AUC | 0.9548 | +| AUC | 0.955 | ## Optimizations | Optimization | Value | diff --git a/models/anomaly_detection/micronet_small/tflite_int8/definition.yaml b/models/anomaly_detection/micronet_small/tflite_int8/definition.yaml index d64ea2b..efe4aed 100644 --- a/models/anomaly_detection/micronet_small/tflite_int8/definition.yaml +++ b/models/anomaly_detection/micronet_small/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: DCASE 2020 Task 2 Slide rail: - AUC: 0.9548 + AUC: 0.955 description: This is a fully quantized version (asymmetrical int8) of the MicroNet Small model developed by Arm, from the MicroNets paper. It is trained on the 'slide rail' task from http://dcase.community/challenge2020/task-unsupervised-detection-of-anomalous-sounds. diff --git a/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/README.md b/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/README.md index 6b55b01..e94eee7 100644 --- a/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/README.md +++ b/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/README.md @@ -39,7 +39,7 @@ Dataset: ILSVRC 2012 | Metric | Value | |--------|-------| -| Top 1 Accuracy | 69.68 | +| Top 1 Accuracy | 0.697 | ## Optimizations | Optimization | Value | diff --git a/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/definition.yaml b/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/definition.yaml index 960b028..ad32dd3 100644 --- a/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/definition.yaml +++ b/models/image_classification/mobilenet_v2_1.0_224/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: ILSVRC 2012: - top-1-accuracy: '69.68' + top-1-accuracy: 0.697 description: "INT8 quantised version of MobileNet v2 model. Trained on ImageNet." license: - Apache-2.0 diff --git a/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/README.md b/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/README.md index 01d6c8a..380b34f 100644 --- a/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/README.md +++ b/models/image_classification/mobilenet_v2_1.0_224/tflite_uint8/README.md @@ -25,7 +25,7 @@ A guide on how to deploy this model using the Arm NN SDK can be found [here](htt ## Performance | Platform | Optimized | |----------|:---------:| -| Cortex-A |:heavy_check_mark: | +| Cortex-A |:heavy_multiplication_x: | | Cortex-M |:heavy_multiplication_x: | | Mali GPU |:heavy_check_mark: | | Ethos U |:heavy_check_mark: | diff --git a/models/keyword_spotting/cnn_large/tflite_int8/README.md b/models/keyword_spotting/cnn_large/tflite_int8/README.md index 43b7cfd..d36c58c 100644 --- a/models/keyword_spotting/cnn_large/tflite_int8/README.md +++ b/models/keyword_spotting/cnn_large/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 92.92% | +| Accuracy | 0.929 | ## Performance | Platform | Optimized | @@ -42,7 +42,6 @@ Dataset: Google Speech Commands Test Set * :heavy_multiplication_x: - Will not run on this platform. - ## Optimizations | Optimization | Value | |-----------------|---------| diff --git a/models/keyword_spotting/cnn_large/tflite_int8/definition.yaml b/models/keyword_spotting/cnn_large/tflite_int8/definition.yaml index fe584ba..fad5eb3 100644 --- a/models/keyword_spotting/cnn_large/tflite_int8/definition.yaml +++ b/models/keyword_spotting/cnn_large/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 92.92% + Accuracy: 0.929 description: 'This is a fully quantized version (asymmetrical int8) of the CNN Large model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/cnn_medium/tflite_int8/README.md b/models/keyword_spotting/cnn_medium/tflite_int8/README.md index ba76824..0ccdf5c 100644 --- a/models/keyword_spotting/cnn_medium/tflite_int8/README.md +++ b/models/keyword_spotting/cnn_medium/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 91.33% | +| Accuracy | 0.913 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/cnn_medium/tflite_int8/definition.yaml b/models/keyword_spotting/cnn_medium/tflite_int8/definition.yaml index 4cd5aea..f5a4b0b 100644 --- a/models/keyword_spotting/cnn_medium/tflite_int8/definition.yaml +++ b/models/keyword_spotting/cnn_medium/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 91.33% + Accuracy: 0.913 description: 'This is a fully quantized version (asymmetrical int8) of the CNN Medium model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/cnn_small/tflite_int8/README.md b/models/keyword_spotting/cnn_small/tflite_int8/README.md index 55299b0..1a8098b 100644 --- a/models/keyword_spotting/cnn_small/tflite_int8/README.md +++ b/models/keyword_spotting/cnn_small/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 91.41% | +| Accuracy | 0.914 | ## Performance | Platform | Optimized | @@ -41,8 +41,6 @@ Dataset: Google Speech Commands Test Set * :heavy_check_mark: - Will run on this platform. * :heavy_multiplication_x: - Will not run on this platform. - - ## Optimizations | Optimization | Value | |-----------------|---------| diff --git a/models/keyword_spotting/cnn_small/tflite_int8/definition.yaml b/models/keyword_spotting/cnn_small/tflite_int8/definition.yaml index b6f10d3..bf73a6a 100644 --- a/models/keyword_spotting/cnn_small/tflite_int8/definition.yaml +++ b/models/keyword_spotting/cnn_small/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 91.41% + Accuracy: 0.914 description: 'This is a fully quantized version (asymmetrical int8) of the CNN Small model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/dnn_large/tflite_int8/README.md b/models/keyword_spotting/dnn_large/tflite_int8/README.md index a47c182..a65b295 100644 --- a/models/keyword_spotting/dnn_large/tflite_int8/README.md +++ b/models/keyword_spotting/dnn_large/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 86.28% | +| Accuracy | 0.863 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/dnn_large/tflite_int8/definition.yaml b/models/keyword_spotting/dnn_large/tflite_int8/definition.yaml index 3549ad3..7731163 100644 --- a/models/keyword_spotting/dnn_large/tflite_int8/definition.yaml +++ b/models/keyword_spotting/dnn_large/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 86.28% + Accuracy: 0.863 description: 'This is a fully quantized version (asymmetrical int8) of the DNN Large model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/dnn_medium/tflite_int8/README.md b/models/keyword_spotting/dnn_medium/tflite_int8/README.md index 05970e7..fcb6e2f 100644 --- a/models/keyword_spotting/dnn_medium/tflite_int8/README.md +++ b/models/keyword_spotting/dnn_medium/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 84.64% | +| Accuracy | 0.846 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/dnn_medium/tflite_int8/definition.yaml b/models/keyword_spotting/dnn_medium/tflite_int8/definition.yaml index cc80108..b963bc7 100644 --- a/models/keyword_spotting/dnn_medium/tflite_int8/definition.yaml +++ b/models/keyword_spotting/dnn_medium/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 84.64% + Accuracy: 0.846 description: 'This is a fully quantized version (asymmetrical int8) of the DNN Medium model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/dnn_small/tflite_int8/README.md b/models/keyword_spotting/dnn_small/tflite_int8/README.md index 54ac9db..6ff5897 100644 --- a/models/keyword_spotting/dnn_small/tflite_int8/README.md +++ b/models/keyword_spotting/dnn_small/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 82.70% | +| Accuracy | 0.827 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/dnn_small/tflite_int8/definition.yaml b/models/keyword_spotting/dnn_small/tflite_int8/definition.yaml index b471b25..891a321 100644 --- a/models/keyword_spotting/dnn_small/tflite_int8/definition.yaml +++ b/models/keyword_spotting/dnn_small/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 82.70% + Accuracy: 0.827 description: 'This is a fully quantized version (asymmetrical int8) of the DNN Small model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/README.md b/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/README.md index 91d6530..0643dd8 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/README.md +++ b/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/README.md @@ -39,7 +39,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Top 1 Accuracy | 0.9495 | +| Top 1 Accuracy | 0.950 | ## Optimizations | Optimization | Value | diff --git a/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/definition.yaml b/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/definition.yaml index 5b23bf1..f9c2303 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/definition.yaml +++ b/models/keyword_spotting/ds_cnn_large/tflite_clustered_fp32/definition.yaml @@ -1,6 +1,6 @@ benchmark: SpeechCommands: - top_1_accuracy: 0.9495 + top_1_accuracy: 0.950 description: 'This is a clustered (32 clusters, kmeans++ centroid initialization) and retrained (fine-tuned) FP32 version of the DS-CNN Large model developed by Arm from the Hello Edge paper. Code for the original DS-CNN implementation can be found diff --git a/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/README.md b/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/README.md index 55e1a1b..3e859ed 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/README.md +++ b/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/README.md @@ -25,7 +25,7 @@ Code to recreate this model can be found here: https://github.com/ARM-software/M ## Performance | Platform | Optimized | |----------|:---------:| -| Cortex-A |:heavy_check_mark: | +| Cortex-A |:heavy_multiplication_x: | | Cortex-M |:heavy_check_mark: | | Mali GPU |:heavy_check_mark: | | Ethos U |:heavy_check_mark: | @@ -39,7 +39,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Top 1 Accuracy | 0.9401 | +| Top 1 Accuracy | 0.940 | ## Optimizations | Optimization | Value | diff --git a/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/definition.yaml b/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/definition.yaml index 0003d0d..3d65144 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/definition.yaml +++ b/models/keyword_spotting/ds_cnn_large/tflite_clustered_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: SpeechCommands: - top_1_accuracy: 0.9401 + top_1_accuracy: 0.940 description: 'This is a clustered (32 clusters, kmeans++ centroid initialization), retrained (fine-tuned) and fully quantized version (INT8) of the DS-CNN Large model developed by Arm from the Hello Edge paper. Code for the original DS-CNN implementation diff --git a/models/keyword_spotting/ds_cnn_large/tflite_int8/README.md b/models/keyword_spotting/ds_cnn_large/tflite_int8/README.md index 472e8ec..b95cf79 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_int8/README.md +++ b/models/keyword_spotting/ds_cnn_large/tflite_int8/README.md @@ -27,12 +27,12 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 94.58% | +| Accuracy | 0.946 | ## Performance | Platform | Optimized | |----------|:---------:| -| Cortex-A |:heavy_check_mark: | +| Cortex-A |:heavy_multiplication_x: | | Cortex-M |:heavy_check_mark: | | Mali GPU |:heavy_check_mark: | | Ethos U |:heavy_check_mark: | diff --git a/models/keyword_spotting/ds_cnn_large/tflite_int8/definition.yaml b/models/keyword_spotting/ds_cnn_large/tflite_int8/definition.yaml index 7588209..b918700 100644 --- a/models/keyword_spotting/ds_cnn_large/tflite_int8/definition.yaml +++ b/models/keyword_spotting/ds_cnn_large/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 94.58% + Accuracy: 0.946 description: 'This is a fully quantized version (asymmetrical int8) of the DS-CNN Large model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/ds_cnn_medium/tflite_int8/README.md b/models/keyword_spotting/ds_cnn_medium/tflite_int8/README.md index 4566e4a..07d6bcf 100644 --- a/models/keyword_spotting/ds_cnn_medium/tflite_int8/README.md +++ b/models/keyword_spotting/ds_cnn_medium/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 93.35% | +| Accuracy | 0.934 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/ds_cnn_medium/tflite_int8/definition.yaml b/models/keyword_spotting/ds_cnn_medium/tflite_int8/definition.yaml index 1caff16..461756b 100644 --- a/models/keyword_spotting/ds_cnn_medium/tflite_int8/definition.yaml +++ b/models/keyword_spotting/ds_cnn_medium/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 93.35% + Accuracy: 0.934 description: 'This is a fully quantized version (asymmetrical int8) of the DS-CNN Medium model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/ds_cnn_small/tflite_int8/README.md b/models/keyword_spotting/ds_cnn_small/tflite_int8/README.md index 097875e..dc2ffec 100644 --- a/models/keyword_spotting/ds_cnn_small/tflite_int8/README.md +++ b/models/keyword_spotting/ds_cnn_small/tflite_int8/README.md @@ -27,7 +27,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 93.35% | +| Accuracy | 0.934 | ## Performance | Platform | Optimized | diff --git a/models/keyword_spotting/ds_cnn_small/tflite_int8/definition.yaml b/models/keyword_spotting/ds_cnn_small/tflite_int8/definition.yaml index 3a529e7..7135f43 100644 --- a/models/keyword_spotting/ds_cnn_small/tflite_int8/definition.yaml +++ b/models/keyword_spotting/ds_cnn_small/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 93.35% + Accuracy: 0.934 description: 'This is a fully quantized version (asymmetrical int8) of the DS-CNN Small model developed by Arm, with training checkpoints, from the Hello Edge paper. Code to recreate this model can be found here: https://github.com/ARM-software/ML-examples/tree/master/tflu-kws-cortex-m' diff --git a/models/keyword_spotting/micronet_large/tflite_int8/README.md b/models/keyword_spotting/micronet_large/tflite_int8/README.md index 4cacf2f..2e3dc16 100644 --- a/models/keyword_spotting/micronet_large/tflite_int8/README.md +++ b/models/keyword_spotting/micronet_large/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 96.48% | +| Accuracy | 0.965 | ## Optimizations | Optimization | Value | diff --git a/models/keyword_spotting/micronet_large/tflite_int8/definition.yaml b/models/keyword_spotting/micronet_large/tflite_int8/definition.yaml index d7ff34a..af8e136 100644 --- a/models/keyword_spotting/micronet_large/tflite_int8/definition.yaml +++ b/models/keyword_spotting/micronet_large/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 96.48% + Accuracy: 0.965 description: This is a fully quantized version (asymmetrical int8) of the MicroNet Large model developed by Arm, from the MicroNets paper. This model is trained on the 'Google Speech Commands' dataset. diff --git a/models/keyword_spotting/micronet_medium/tflite_int8/README.md b/models/keyword_spotting/micronet_medium/tflite_int8/README.md index 8cfa7f2..9c7db99 100644 --- a/models/keyword_spotting/micronet_medium/tflite_int8/README.md +++ b/models/keyword_spotting/micronet_medium/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 95.77% | +| Accuracy | 0.958 | ## Optimizations | Optimization | Value | diff --git a/models/keyword_spotting/micronet_medium/tflite_int8/definition.yaml b/models/keyword_spotting/micronet_medium/tflite_int8/definition.yaml index 60e49d5..d4f876a 100644 --- a/models/keyword_spotting/micronet_medium/tflite_int8/definition.yaml +++ b/models/keyword_spotting/micronet_medium/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 95.77% + Accuracy: 0.958 description: This is a fully quantized version (asymmetrical int8) of the MicroNet Medium model developed by Arm, from the MicroNets paper. This model is trained on the 'Google Speech Commands' dataset. diff --git a/models/keyword_spotting/micronet_small/tflite_int8/README.md b/models/keyword_spotting/micronet_small/tflite_int8/README.md index aaa3b26..35ff26e 100644 --- a/models/keyword_spotting/micronet_small/tflite_int8/README.md +++ b/models/keyword_spotting/micronet_small/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Google Speech Commands Test Set | Metric | Value | |--------|-------| -| Accuracy | 95.32% | +| Accuracy | 0.953 | ## Optimizations | Optimization | Value | diff --git a/models/keyword_spotting/micronet_small/tflite_int8/definition.yaml b/models/keyword_spotting/micronet_small/tflite_int8/definition.yaml index 4217caa..344bfc4 100644 --- a/models/keyword_spotting/micronet_small/tflite_int8/definition.yaml +++ b/models/keyword_spotting/micronet_small/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Google Speech Commands test set: - Accuracy: 95.32% + Accuracy: 0.953 description: This is a fully quantized version (asymmetrical int8) of the MicroNet Small model developed by Arm, from the MicroNets paper. This model is trained on the 'Google Speech Commands' dataset. diff --git a/models/object_detection/ssd_mobilenet_v1/tflite_fp32/README.md b/models/object_detection/ssd_mobilenet_v1/tflite_fp32/README.md index 656b25c..34aefb9 100644 --- a/models/object_detection/ssd_mobilenet_v1/tflite_fp32/README.md +++ b/models/object_detection/ssd_mobilenet_v1/tflite_fp32/README.md @@ -30,7 +30,7 @@ Dataset: Coco Validation 2017 | Metric | Value | |--------|-------| -| mAP | 0.21 | +| mAP | 0.210 | ## Performance | Platform | Optimized | diff --git a/models/object_detection/ssd_mobilenet_v1/tflite_fp32/definition.yaml b/models/object_detection/ssd_mobilenet_v1/tflite_fp32/definition.yaml index a2ef199..87ed1e3 100644 --- a/models/object_detection/ssd_mobilenet_v1/tflite_fp32/definition.yaml +++ b/models/object_detection/ssd_mobilenet_v1/tflite_fp32/definition.yaml @@ -1,6 +1,6 @@ benchmark: coco_validation_2017: - mAP: 0.21 + mAP: 0.210 description: SSD MobileNet v1 is a object detection network, that localizes and identifies objects in an input image. This is a TF Lite floating point version that takes a 300x300 input image and outputs detections for this image. This model is trained diff --git a/models/object_detection/ssd_mobilenet_v1/tflite_int8/definition.yaml b/models/object_detection/ssd_mobilenet_v1/tflite_int8/definition.yaml index af99a1f..07a6a88 100644 --- a/models/object_detection/ssd_mobilenet_v1/tflite_int8/definition.yaml +++ b/models/object_detection/ssd_mobilenet_v1/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: COCO 2017 Validation: - mAP: '0.234' + mAP: 0.234 description: SSD MobileNet v1 is a object detection network, that localizes and identifies objects in an input image. This is a TF Lite quantized version that takes a 300x300 input image and outputs detections for this image. This model is converted from diff --git a/models/object_detection/ssd_mobilenet_v1/tflite_uint8/README.md b/models/object_detection/ssd_mobilenet_v1/tflite_uint8/README.md index d74c212..66431d8 100644 --- a/models/object_detection/ssd_mobilenet_v1/tflite_uint8/README.md +++ b/models/object_detection/ssd_mobilenet_v1/tflite_uint8/README.md @@ -27,12 +27,12 @@ Dataset: Coco Validation 2017 | Metric | Value | |--------|-------| -| mAP | 0.18 | +| mAP | 0.180 | ## Performance | Platform | Optimized | |----------|:---------:| -| Cortex-A |:heavy_check_mark: | +| Cortex-A |:heavy_multiplication_x: | | Cortex-M |:heavy_multiplication_x: | | Mali GPU |:heavy_check_mark: | | Ethos U |:heavy_multiplication_x: | diff --git a/models/object_detection/ssd_mobilenet_v1/tflite_uint8/definition.yaml b/models/object_detection/ssd_mobilenet_v1/tflite_uint8/definition.yaml index f2818fe..6fb133f 100644 --- a/models/object_detection/ssd_mobilenet_v1/tflite_uint8/definition.yaml +++ b/models/object_detection/ssd_mobilenet_v1/tflite_uint8/definition.yaml @@ -1,6 +1,6 @@ benchmark: coco_validation_2017: - mAP: 0.18 + mAP: 0.180 description: SSD MobileNet v1 is a object detection network, that localizes and identifies objects in an input image. This is a TF Lite quantized version that takes a 300x300 input image and outputs detections for this image. This model is trained and quantized diff --git a/models/speech_recognition/wav2letter/tflite_int8/README.md b/models/speech_recognition/wav2letter/tflite_int8/README.md index 31d4290..4916c88 100644 --- a/models/speech_recognition/wav2letter/tflite_int8/README.md +++ b/models/speech_recognition/wav2letter/tflite_int8/README.md @@ -24,7 +24,7 @@ Dataset: Librispeech | Metric | Value | |--------|-------| -| Ler | 0.08771 | +| Ler | 0.0877 | ## Performance | Platform | Optimized | diff --git a/models/speech_recognition/wav2letter/tflite_int8/definition.yaml b/models/speech_recognition/wav2letter/tflite_int8/definition.yaml index 01e49a4..9787caf 100644 --- a/models/speech_recognition/wav2letter/tflite_int8/definition.yaml +++ b/models/speech_recognition/wav2letter/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: LibriSpeech: - LER: 0.08771 + LER: 0.0877 description: Wav2letter is a convolutional speech recognition neural network. This implementation was created by Arm and quantized to the INT8 datatype. license: diff --git a/models/speech_recognition/wav2letter/tflite_pruned_int8/README.md b/models/speech_recognition/wav2letter/tflite_pruned_int8/README.md index 0cd3255..6c524c4 100644 --- a/models/speech_recognition/wav2letter/tflite_pruned_int8/README.md +++ b/models/speech_recognition/wav2letter/tflite_pruned_int8/README.md @@ -39,7 +39,7 @@ Dataset: LibriSpeech | Metric | Value | |--------|-------| -| LER | 0.07831 | +| LER | 0.0783 | ## Optimizations | Optimization | Value | diff --git a/models/speech_recognition/wav2letter/tflite_pruned_int8/definition.yaml b/models/speech_recognition/wav2letter/tflite_pruned_int8/definition.yaml index a2f0ce0..8b417a2 100644 --- a/models/speech_recognition/wav2letter/tflite_pruned_int8/definition.yaml +++ b/models/speech_recognition/wav2letter/tflite_pruned_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: LibriSpeech: - LER: 0.07831443101167679 + LER: 0.0783 description: Wav2letter is a convolutional speech recognition neural network. This implementation was created by Arm, pruned to 50% sparisty, fine-tuned and quantized using the TensorFlow Model Optimization Toolkit. diff --git a/models/visual_wake_words/micronet_vww2/tflite_int8/README.md b/models/visual_wake_words/micronet_vww2/tflite_int8/README.md index 03dc1e4..d6c3694 100644 --- a/models/visual_wake_words/micronet_vww2/tflite_int8/README.md +++ b/models/visual_wake_words/micronet_vww2/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Visual Wake Words | Metric | Value | |--------|-------| -| Accuracy | 76.8 | +| Accuracy | 0.768 | ## Optimizations | Optimization | Value | diff --git a/models/visual_wake_words/micronet_vww2/tflite_int8/definition.yaml b/models/visual_wake_words/micronet_vww2/tflite_int8/definition.yaml index 4b2a1d1..29f1b47 100644 --- a/models/visual_wake_words/micronet_vww2/tflite_int8/definition.yaml +++ b/models/visual_wake_words/micronet_vww2/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Visual Wake Words: - accuracy: 76.8 + accuracy: 0.768 description: 'This is a fully quantized version (asymmetrical int8) of the MicroNet VWW-2 model developed by Arm, from the MicroNets paper. It is trained on the ''Visual Wake Words'' dataset, more information can be found here: https://arxiv.org/pdf/1906.05721.pdf.' diff --git a/models/visual_wake_words/micronet_vww3/tflite_int8/README.md b/models/visual_wake_words/micronet_vww3/tflite_int8/README.md index 9cebb50..b31788c 100644 --- a/models/visual_wake_words/micronet_vww3/tflite_int8/README.md +++ b/models/visual_wake_words/micronet_vww3/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Visual Wake Words | Metric | Value | |--------|-------| -| Accuracy | 85.53723 | +| Accuracy | 0.855 | ## Optimizations | Optimization | Value | diff --git a/models/visual_wake_words/micronet_vww3/tflite_int8/definition.yaml b/models/visual_wake_words/micronet_vww3/tflite_int8/definition.yaml index 1527e3a..a3ca36f 100644 --- a/models/visual_wake_words/micronet_vww3/tflite_int8/definition.yaml +++ b/models/visual_wake_words/micronet_vww3/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Visual Wake Words: - Accuracy: 85.53723 + Accuracy: 0.855 description: 'This is a fully quantized version (asymmetrical int8) of the MicroNet VWW-3 model developed by Arm, from the MicroNets paper. It is trained on the ''Visual Wake Words'' dataset, more information can be found here: https://arxiv.org/pdf/1906.05721.pdf.' diff --git a/models/visual_wake_words/micronet_vww4/tflite_int8/README.md b/models/visual_wake_words/micronet_vww4/tflite_int8/README.md index 5de027f..ce18021 100644 --- a/models/visual_wake_words/micronet_vww4/tflite_int8/README.md +++ b/models/visual_wake_words/micronet_vww4/tflite_int8/README.md @@ -36,7 +36,7 @@ Dataset: Visual Wake Words | Metric | Value | |--------|-------| -| Accuracy | 82.19682 | +| Accuracy | 0.822| ## Optimizations | Optimization | Value | diff --git a/models/visual_wake_words/micronet_vww4/tflite_int8/definition.yaml b/models/visual_wake_words/micronet_vww4/tflite_int8/definition.yaml index 7308d59..c4b560b 100644 --- a/models/visual_wake_words/micronet_vww4/tflite_int8/definition.yaml +++ b/models/visual_wake_words/micronet_vww4/tflite_int8/definition.yaml @@ -1,6 +1,6 @@ benchmark: Visual Wake Words: - Accuracy: '82.19682' + Accuracy: 0.822 description: 'This is a fully quantized version (asymmetrical int8) of the MicroNet VWW-4 model developed by Arm, from the MicroNets paper. It is trained on the ''Visual Wake Words'' dataset, more information can be found here: https://arxiv.org/pdf/1906.05721.pdf.'