Update to 1.1.10 Release

This release is compatible with DeepStream SDK 6.4

Ubuntu 22.04
Python 3.10
DeepStream SDK 6.4

Features:
- New API for alloc_nvds_event_msg_meta() - see deepstream-test4 and bindschema.cpp for reference
- Tracker meta data type names have been updated with DS 6.4 release - see deepstream-test2 for reference
- deepstream-test4 has been updated with information on how to run with MQTT protocol adaptor
- All app configurations have been updated to use newest model engines that ship with DS 6.4 and latest TAO toolkit
- deepstream-test1-rtsp-out has been updated to demonstrate support for Orin Nano by adding an option to use software encoder

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "3rdparty/pybind11"]
 	path = 3rdparty/pybind11
 	url = https://github.com/pybind/pybind11.git
-[submodule "3rdparty/gst-python"]
-	path = 3rdparty/gst-python
-	url = https://github.com/GStreamer/gst-python.git
+[submodule "gstreamer"]
+	path = 3rdparty/gstreamer
+	url = https://github.com/GStreamer/gstreamer.git

FAQ.md

@@ -1,6 +1,7 @@
 # Frequently Asked Questions and Troubleshooting Guide
 
 - [Frequently Asked Questions and Troubleshooting Guide](#frequently-asked-questions-and-troubleshooting-guide)
+    - [App causes Jetson Orin Nano to freeze and reboot](#app-causes-jetson-orin-nano-to-freeze-and-reboot)
     - [Using new gst-nvstreammux](#using-new-gst-nvstreammux)
     - [Git clone fails due to Gstreamer repo access error](#git-clone-fails-due-to-gstreamer-repo-access-error)
     - [Application fails to work with mp4 stream](#application-fails-to-work-with-mp4-stream)
@@ -13,6 +14,18 @@
     - [Triton container problems with multi-GPU setup](#triton-container-problems-with-multi-gpu-setup)
     - [ModuleNotFoundError: No module named 'pyds'](#modulenotfounderror-no-module-named-pyds)
 
+<a name="faq11"></a>
+### App causes Jetson Orin Nano to freeze and reboot
+Most of the DeepStream Python apps are written to use hardware encoders. The Jetson Orin Nano does not have any hardware encoders, so the app must be modified to use software encoders in the pipeline. Please see [deepstream-test1-rtsp-out](./apps/deepstream-test1-rtsp-out/deepstream_test_1_rtsp_out.py):
+```python
+if codec == "H264":
+  encoder = Gst.ElementFactory.make("nvv4l2h264enc", "encoder")
+  if enc_type == 0: # HW encoder
+      encoder = Gst.ElementFactory.make("nvv4l2h264enc", "encoder")
+  else: # SW encoder
+      encoder = Gst.ElementFactory.make("x264enc", "encoder")
+```
+
 <a name="faq10"></a>
 ### Using new gst-nvstreammux
 Most DeepStream Python apps are written to use the default nvstreammux and have lines of code written to set nvstreammux properties which are deprecated in the new gst-nvstreammux. See the [DeepStream documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux2.html) for more information on the new gst-nvstreammux plugin. To use the new nvstreammux, set the `USE_NEW_NVSTREAMMUX` environment variable before running the app. For example:
@@ -152,5 +165,5 @@ The pyds wheel installs the pyds.so library where all the pip packages are store
 
 Command to install the pyds wheel is:
 ```bash
-   $ pip3 install ./pyds-1.1.8-py3-none*.whl
+   $ pip3 install ./pyds-1.1.10-py3-none*.whl
 ```

HOWTO.md

@@ -15,10 +15,10 @@ This guide provides resources for DeepStream application development in Python.
 <a name="prereqs"></a>
 ## Prerequisites
 
-* Ubuntu 20.04
-* [DeepStream SDK 6.3](https://developer.nvidia.com/deepstream-download) or later
-* Python 3.8
-* [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.16.2
+* Ubuntu 22.04
+* [DeepStream SDK 6.4](https://developer.nvidia.com/deepstream-download) or later
+* Python 3.10
+* [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.20.3
 
 Gst python should be already installed on Jetson.  
 If missing, install with the following steps:
@@ -84,12 +84,12 @@ Memory for MetaData is shared by the Python and C/C++ code paths. For example, a
 
 #### Allocations
 
-When MetaData objects are allocated in Python, an allocation function is provided by the bindings to ensure proper memory ownership of the object. If the constructor is used, the the object will be claimed by the garbage collector when its Python references terminate. However, the object will still need to be accessed by C/C++ code downstream, and therefore must persist beyond those Python references.  
+When MetaData objects are allocated in Python, an allocation function is provided by the bindings to ensure proper memory ownership of the object. If the constructor is used, the object will be claimed by the garbage collector when its Python references terminate. However, the object will still need to be accessed by C/C++ code downstream, and therefore must persist beyond those Python references.  
 
 Example:
 To allocate an NvDsEventMsgMeta instance, use this:  
 ```python
-msg_meta = pyds.alloc_nvds_event_msg_meta() # get reference to allocated instance without claiming memory ownership  
+msg_meta = pyds.alloc_nvds_event_msg_meta(user_event_meta) # get reference to allocated instance without claiming memory ownership  
 ```
 NOT this:  
 ```python
@@ -173,16 +173,21 @@ frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
 
 Custom MetaData added to NvDsUserMeta require custom copy and release functions. The MetaData library relies on these custom functions to perform deep-copy of the custom structure, and free allocated resources. These functions are registered as callback function pointers in the NvDsUserMeta structure.
 
-Callback functions are registered using these functions:  
+##### Deprecation Warning
+Previously, Callback functions were registered using these functions:  
 ```python
 pyds.set_user_copyfunc(NvDsUserMeta_instance, copy_function)
 pyds.set_user_releasefunc(NvDsUserMeta_instance, free_func)
 ```
 
-*NOTE*: Callbacks need to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
+These are now DEPRECATED and are replaced by similar implementation in the binding itself. These are event_msg_meta_copy_func() and event_msg_meta_release_func() respectively. These can be found inside [bindschema.cpp](bindings/src/bindschema.cpp)
+
+##### Deprecation Warning
+*NOTE*: Previously, callbacks needed to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
 ```pyds.unset_callback_funcs()```  
+These callbacks are automatically set inside the alloc_nvds_event_msg_meta() function and should NOT be set from the python application (e.g. deepstream-test4)
 
-See the deepstream-test4 sample application for an example of callback registration and unregistration.  
+The deepstream-test4 sample application has been updated to show an example of removal of these callback registration and unregistration.
 
 Limitation: the bindings library currently only supports a single set of callback functions for each application. The last registered function will be used.  
 

README.md

@@ -2,11 +2,11 @@
 
 This repository contains Python bindings and sample applications for the [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk).  
 
-SDK version supported: 6.3
+SDK version supported: 6.4
 
-<b>The bindings sources along with build instructions are now available under [bindings](bindings)! We now include a [guide](bindings/BINDINGSGUIDE.md) for contributing to bindings and another [guide](bindings/CUSTOMUSERMETAGUIDE.md) for advanced use-cases such as writing bindings for custom data structures.</b>
+<b>This release only supports Ubuntu 22.04 for DeepStreamSDK 6.4 with Python 3.10 and [gst-python](3rdparty/gst-python/) 1.20.3! Ubuntu 20.04 for DeepStreamSDK 6.3 with Python 3.8 support is NOW DEPRECATED</b>
 
-<b>This release only supports Ubuntu 20.04 for DeepStreamSDK 6.3 with Python 3.8 and [gst-python](3rdparty/gst-python/) 1.16.2! Ubuntu 18.04 support is now deprecrated.</b>
+The bindings sources along with build instructions are available under [bindings](bindings)! We include one [guide](bindings/BINDINGSGUIDE.md) for contributing to bindings and another [guide](bindings/CUSTOMUSERMETAGUIDE.md) for advanced use-cases such as writing bindings for custom data structures.
 
 Download the latest release package complete with bindings and sample applications from the [release section](../../releases).  
 
@@ -28,6 +28,9 @@ Python [bindings](bindings) are provided as part of this repository. This module
 
 These bindings support a Python interface to the MetaData structures and functions. Usage of this interface is documented in the [HOW-TO Guide](HOWTO.md) and demonstrated in the sample applications.  
 
+### Python Bindings Breaking API Change
+The binding for function alloc_nvds_event_msg_meta() now expects a NvDsUserMeta pointer which the NvDsEventMsgMeta is associated with. Please refer to [deepstream-test4](apps/deepstream-test4) and [bindschema.cpp](bindings/src/bindschema.cpp) for reference.
+
 <a name="sample_applications"></a>
 ## Sample Applications
 
@@ -41,20 +44,20 @@ To run the sample applications or write your own, please consult the [HOW-TO Gui
 </p>
 
 We currently provide the following sample applications:
-* [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline - now also demonstrates support for new nvstreammux
-* [deepstream-test2](apps/deepstream-test2) -- 4-class object detection, tracking and attribute classification pipeline
-* [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection - now also supports triton inference server, no-display mode, file-loop and silent mode
-* [deepstream-test4](apps/deepstream-test4) -- msgbroker for sending analytics results to the cloud
+* [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline, also demonstrates support for new nvstreammux
+* **UPDATED** [deepstream-test2](apps/deepstream-test2) -- 4-class object detection, tracking and attribute classification pipeline - now uses new names for tracker meta data types in DS 6.4
+* [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection, also supports triton inference server, no-display mode, file-loop and silent mode
+* **UPDATED** [deepstream-test4](apps/deepstream-test4) -- msgbroker for sending analytics results to the cloud - now supports MQTT protocol adaptor
 * [deepstream-imagedata-multistream](apps/deepstream-imagedata-multistream) -- multi-stream pipeline with access to image buffers
 * [deepstream-ssd-parser](apps/deepstream-ssd-parser) -- SSD model inference via Triton server with output parsing in Python
 * [deepstream-test1-usbcam](apps/deepstream-test1-usbcam) -- deepstream-test1 pipeline with USB camera input
-* [deepstream-test1-rtsp-out](apps/deepstream-test1-rtsp-out) -- deepstream-test1 pipeline with RTSP output
+* **UPDATED** [deepstream-test1-rtsp-out](apps/deepstream-test1-rtsp-out) -- deepstream-test1 pipeline with RTSP output - now demonstrates adding software encoder option to support Jetson Orin Nano
 * [deepstream-opticalflow](apps/deepstream-opticalflow) -- optical flow and visualization pipeline with flow vectors returned in NumPy array
 * [deepstream-segmentation](apps/deepstream-segmentation) -- segmentation and visualization pipeline with segmentation mask returned in NumPy array
 * [deepstream-nvdsanalytics](apps/deepstream-nvdsanalytics) -- multistream pipeline with analytics plugin
 * [runtime_source_add_delete](apps/runtime_source_add_delete) -- add/delete source streams at runtime
 * [deepstream-imagedata-multistream-redaction](apps/deepstream-imagedata-multistream-redaction) -- multi-stream pipeline with face detection and redaction
-* <b>UPDATED</b> [deepstream-rtsp-in-rtsp-out](apps/deepstream-rtsp-in-rtsp-out) -- multi-stream pipeline with RTSP input/output -- now takes new command line argument "--rtsp-ts" for configuring the RTSP source to attach the timestamp rather than the streammux
+* [deepstream-rtsp-in-rtsp-out](apps/deepstream-rtsp-in-rtsp-out) -- multi-stream pipeline with RTSP input/output - has command line option "--rtsp-ts" for configuring the RTSP source to attach the timestamp rather than the streammux
 * [deepstream-preprocess-test](apps/deepstream-preprocess-test) -- multi-stream pipeline using nvdspreprocess plugin with custom ROIs
 * [deepstream-demux-multi-in-multi-out](apps/deepstream-demux-multi-in-multi-out) -- multi-stream pipeline using nvstreamdemux plugin to generated separate buffer outputs
 * [deepstream-imagedata-multistream-cupy](apps/deepstream-imagedata-multistream-cupy) -- access imagedata buffer from GPU in a multistream source as CuPy array - x86 only

apps/README

@@ -19,9 +19,9 @@
 DeepStream SDK Python Bindings
 ================================================================================
 Setup pre-requisites:
-- Ubuntu 20.04
-- NVIDIA DeepStream SDK 6.3
-- Python 3.8
+- Ubuntu 22.04
+- NVIDIA DeepStream SDK 6.4
+- Python 3.10
 - Gst-python
 
 --------------------------------------------------------------------------------
@@ -36,13 +36,13 @@ Package Contents
 Installing Pre-requisites:
 --------------------------------------------------------------------------------
 
-DeepStream SDK 6.3
+DeepStream SDK 6.4
 --------------------
 Download and install from https://developer.nvidia.com/deepstream-download
 
-Python 3.8
+Python 3.10
 ----------
-Should be already installed with Ubuntu 20.04
+Should be already installed with Ubuntu 22.04
 
 Gst-python
 ----------
@@ -54,7 +54,7 @@ $ sudo apt install python3-gi python3-dev python3-gst-1.0 -y
 --------------------------------------------------------------------------------
 Running the samples
 --------------------------------------------------------------------------------
-The apps are configured to work from inside the DeepStream SDK 6.3 installation.
+The apps are configured to work from inside the DeepStream SDK 6.4 installation.
 
 Clone the deepstream_python_apps repo under <DeepStream ROOT>/sources:
 $ git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps

apps/deepstream-custom-binding-test/README

@@ -16,8 +16,8 @@
 ################################################################################
 
 Prequisites:
-- DeepStreamSDK 6.3
-- Python 3.8
+- DeepStreamSDK 6.4
+- Python 3.10
 - Gst-python
 
 To run the test app:

apps/deepstream-demux-multi-in-multi-out/README

@@ -16,8 +16,8 @@
 ################################################################################
 
 Prerequisites:
-- DeepStreamSDK 6.3
-- Python 3.8
+- DeepStreamSDK 6.4
+- Python 3.10
 - Gst-python
 
 To run:

apps/deepstream-demux-multi-in-multi-out/deepstream_demux_multi_in_multi_out.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 ################################################################################
-# SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -50,7 +50,7 @@
 PGIE_CLASS_ID_ROADSIGN = 3
 MUXER_OUTPUT_WIDTH = 540
 MUXER_OUTPUT_HEIGHT = 540  # 1080
-MUXER_BATCH_TIMEOUT_USEC = 4000000
+MUXER_BATCH_TIMEOUT_USEC = 33000
 TILED_OUTPUT_WIDTH = 640  # 1280
 TILED_OUTPUT_HEIGHT = 360  # 720
 GST_CAPS_FEATURES_NVMM = "memory:NVMM"
@@ -321,7 +321,7 @@ def main(args, requested_pgie=None, config=None, disable_probe=False):
     streammux.set_property("width", 960)
     streammux.set_property("height", 540)
     streammux.set_property("batch-size", number_sources)
-    streammux.set_property("batched-push-timeout", 4000000)
+    streammux.set_property("batched-push-timeout", MUXER_BATCH_TIMEOUT_USEC)
     pgie.set_property("config-file-path", "ds_demux_pgie_config.txt")
     pgie_batch_size = pgie.get_property("batch-size")
     if pgie_batch_size != number_sources:

apps/deepstream-demux-multi-in-multi-out/ds_demux_pgie_config.txt

@@ -1,5 +1,5 @@
 ################################################################################
-# SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2019-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -26,7 +26,7 @@
 #
 # Optional properties for detectors:
 #   cluster-mode(Default=Group Rectangles), interval(Primary mode only, Default=0)
-#   custom-lib-path
+#   custom-lib-path,
 #   parse-bbox-func-name
 #
 # Mandatory properties for classifiers:
@@ -54,24 +54,30 @@
 
 [property]
 gpu-id=0
-net-scale-factor=0.0039215697906911373
-model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel
-proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt
-model-engine-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
+net-scale-factor=0.00392156862745098
+tlt-model-key=tlt_encode
+tlt-encoded-model=../../../../samples/models/Primary_Detector/resnet18_trafficcamnet.etlt
+model-engine-file=../../../../samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b1_gpu0_int8.engine
 labelfile-path=../../../../samples/models/Primary_Detector/labels.txt
 int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin
 force-implicit-batch-dim=1
-batch-size=1
+batch-size=30
 process-mode=1
 model-color-format=0
+## 0=FP32, 1=INT8, 2=FP16 mode
 network-mode=1
 num-detected-classes=4
 interval=0
 gie-unique-id=1
-output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
+uff-input-order=0
+uff-input-blob-name=input_1
+output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
+#scaling-filter=0
+#scaling-compute-hw=0
 cluster-mode=2
+infer-dims=3;544;960
 
 [class-attrs-all]
 pre-cluster-threshold=0.2
-topk=20
-nms-iou-threshold=0.5
+eps=0.2
+group-threshold=1

apps/deepstream-imagedata-multistream-cupy/README

@@ -16,8 +16,8 @@
 ################################################################################
 
 Prerequisites:
-- DeepStreamSDK 6.3
-- Python 3.8
+- DeepStreamSDK 6.4
+- Python 3.10
 - Gst-python
 - NumPy package
 - OpenCV package

apps/deepstream-imagedata-multistream-cupy/deepstream_imagedata-multistream_cupy.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 ################################################################################
-# SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -45,7 +45,7 @@
 PGIE_CLASS_ID_ROADSIGN = 3
 MUXER_OUTPUT_WIDTH = 1920
 MUXER_OUTPUT_HEIGHT = 1080
-MUXER_BATCH_TIMEOUT_USEC = 4000000
+MUXER_BATCH_TIMEOUT_USEC = 33000
 TILED_OUTPUT_WIDTH = 1920
 TILED_OUTPUT_HEIGHT = 1080
 GST_CAPS_FEATURES_NVMM = "memory:NVMM"
@@ -285,7 +285,7 @@ def main(args):
     streammux.set_property('width', 1920)
     streammux.set_property('height', 1080)
     streammux.set_property('batch-size', number_sources)
-    streammux.set_property('batched-push-timeout', 4000000)
+    streammux.set_property('batched-push-timeout', MUXER_BATCH_TIMEOUT_USEC)
     pgie.set_property('config-file-path', "dstest_imagedata_cupy_config.txt")
     pgie_batch_size = pgie.get_property("batch-size")
     if (pgie_batch_size != number_sources):