Add additional image processing operators for Ascend NPU by utilizing DVPP #3608

MengqingCao · 2023-12-13T09:36:06Z

The user base for Ascend NPU and programming with CANN is increasing rapidly, with a growing number of users joining each day. To facilitate the use of these users, this PR provides more support for Ascend backend operators. All operators this PR offers are using use DVPP as the computational unit. Digital Vision Pre-Processing (DVPP) is an image processing unit built into the Ascend AI processor. Its main functions include image and video encoding/decoding, as well as image cropping and scaling.

The high-frequency operators with NPU as the backend and basic data structure AscendMat has been provided in #3552, while it still lacks many image processing operators. Moreover, only two interpolation algorithms for the resize operator are supported in #3552. In this PR, the bilinear interpolation algorithm and nearest neighbour interpolation algorithm are implemented for the resize operator, as well as the Ascend implementation of the copyMakeBorder operator.

In addition, the serialization of image processing operations is widely used in the preprocessing and post-processing stages of computer vision deep learning methods. Therefore, providing integrated operators is very meaningful for improving the convenience of use for OpenCV and deep learning crossover users. For example, torchvision also provides similar operators: RESIZED_CROP.
Thus, this PR also provides two serialization processing operators: cropResize and cropResizeMakeBorder.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
[N/A] There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

MengqingCao · 2024-01-02T07:27:34Z

@opencv-alalek Happy New Year! This PR relies on CANN backends, could you trigger the test by labelling it? BTW, it may fail until the library it needs is linked. (Link libacl_dvpp_mpi library when building with CANN backend #24720)

The build on win64 is scheduled, and I have no idea with how to deal this.

opencv-alalek · 2024-01-02T07:48:45Z

The build on win64 is scheduled, and I have no idea with how to deal this.

Windows builder worker was paused due to maintenance. Ignore.

I added "cann" label but GHA doesn't launch CANN build configuration due to some reason.

If you want to test both PRs/patches from opencv and opencv_contrib repositories then they must have the same "source" branch name (currently names are different). You need to push branch to opencv under the name dvpp_support.

MengqingCao · 2024-01-02T08:10:32Z

I added "cann" label but GHA doesn't launch CANN build configuration due to some reason.

Thx, are there any errors with the CI machine or bugs on GHA?

If you want to test both PRs/patches from opencv and opencv_contrib repositories then they must have the same "source" branch name (currently names are different). You need to push branch to opencv under the name dvpp_support.

Got it, I will resubmit a PR in opencv repo.

MengqingCao · 2024-01-11T07:02:19Z

Good Day! @opencv-alalek, Thanks for your help, now CANN's CI is working properly and this PR passed all the checks. Could you please help review this PR or assign someone else to review it? This PR is mainly based on #3552 with the new operators utilizing DVPP for Ascend NPU, providing more interpolation methods for resize, adding copyMakeBorder and 2 serialization operators.
Looking forward to suggestions! Thanks!

opencv-alalek · 2024-01-11T07:25:32Z

/cc @fengyuentau

opencv-alalek · 2024-01-11T07:22:59Z

modules/cannops/src/dvpp_call.cpp

+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+#ifndef ENABLE_DVPP_INTERFACE


Do we need this check in .cpp file?

If yes, then I expecting #endif much closer (like modules/cannops/include/opencv2/dvpp_call.hpp)

No, it will be deleted in next commit. Thanks for the correction

fengyuentau · 2024-01-12T14:58:06Z

modules/cannops/src/dvpp_call.cpp

+    holder.clear();
+    return *this;
+}
+void initDvpp() { hi_mpi_sys_init(); }


What devices are supported here? I see from the document that

Atlas 200/300/500 推理产品，不支持该接口

Also it turns out these are from dvpp v2. In dvpp v1, there is no such a thing.

What devices are supported here? I see from the document that

Atlas 200/300/500 推理产品，不支持该接口

Atlas Inference Series products, Atlas 200/500 A2 Inference products and Atlas A2 Training Series products/Atlas 300I A2 Inference products are supported. The detailed information can be found in the official documentation in 功能支持度说明. Thank you for the suggestion, the description about supported devices is very important for users, this information will be clearly clarified in the form of a comment in the next commit.

Also it turns out these are from dvpp v2. In dvpp v1, there is no such a thing.

The Ascend official documentation indicates that the DVPP V2 version of the interface is recommended. DVPP V1 version is only compatible with older versions and will not be evolved in the future, so DVPP V2 is used to follow the future evolution.

fengyuentau

👍

MengqingCao · 2024-01-19T03:46:43Z

Thanks for the review! All the fixes, as well as the cherry-pick have been done now.

1. add the Ascend dvpp calling code 2. add copyMakeBorder, cropResize and cropResizeMakeBorder 3. add resize with INTER_LINEAR and INTER_NEAREST interpolation

MengqingCao · 2024-01-30T07:43:29Z

@asmorkalov Good Day! Should the code in this PR be included in the branch?

MengqingCao · 2024-02-20T02:23:39Z

@asmorkalov Sorry to bother you, could you check if this project is now mergeable?

fengyuentau · 2024-02-20T03:47:28Z

Hello @MengqingCao , could you help resolve comments in opencv-infrastructure/opencv-gha-dockerfile#28?

MengqingCao · 2024-02-20T04:36:14Z

Hello @MengqingCao , could you help resolve comments in opencv-infrastructure/opencv-gha-dockerfile#28?

sure, I'll check it soon.

MengqingCao · 2024-02-29T08:59:14Z

@asmorkalov Sorry to bother you again, could you please review whether this pull request should be included? thx 👍

MengqingCao · 2024-03-25T06:39:00Z

@fengyuentau @asmorkalov Good day! I noticed that the CI passed but was not updated in the build check on this page. Should the CI be retriggered or anything else to do with it?

Add additional image processing operators for Ascend NPU by utilizing DVPP opencv#3608 The user base for [Ascend NPU](https://www.hiascend.com/en/) and programming with CANN is increasing rapidly, with a growing number of users joining each day. To facilitate the use of these users, this PR provides more support for Ascend backend operators. All operators this PR offers are using use DVPP as the computational unit. Digital Vision Pre-Processing (DVPP) is an image processing unit built into the Ascend AI processor. Its main functions include image and video encoding/decoding, as well as image cropping and scaling. The high-frequency operators with NPU as the backend and basic data structure AscendMat has been provided in opencv#3552, while it still lacks many image processing operators. Moreover, only two interpolation algorithms for the resize operator are supported in opencv#3552. In this PR, the bilinear interpolation algorithm and nearest neighbour interpolation algorithm are implemented for the resize operator, as well as the Ascend implementation of the copyMakeBorder operator. In addition, the serialization of image processing operations is widely used in the preprocessing and post-processing stages of computer vision deep learning methods. Therefore, providing integrated operators is very meaningful for improving the convenience of use for OpenCV and deep learning crossover users. For example, torchvision also provides similar operators: [RESIZED_CROP](https://pytorch.org/vision/stable/generated/torchvision.transforms.functional.resized_crop.html?highlight=resizedcrop). Thus, this PR also provides two serialization processing operators: cropResize and cropResizeMakeBorder. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [N/A] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

MengqingCao mentioned this pull request Dec 19, 2023

Link libacl_dvpp_mpi library when building with CANN backend opencv/opencv#24720

Closed

5 tasks

MengqingCao force-pushed the dvpp_support branch 3 times, most recently from 95a6756 to 15bf9bd Compare January 2, 2024 07:03

opencv-alalek added the category: cann label Jan 2, 2024

MengqingCao mentioned this pull request Jan 2, 2024

Link libacl_dvpp_mpi library when building with CANN backend opencv/opencv#24805

Merged

5 tasks

opencv-alalek reviewed Jan 11, 2024

View reviewed changes

fengyuentau reviewed Jan 12, 2024

View reviewed changes

MengqingCao force-pushed the dvpp_support branch from a15b091 to 1386f6b Compare January 15, 2024 07:21

fengyuentau approved these changes Jan 16, 2024

View reviewed changes

opencv-alalek assigned asmorkalov Jan 17, 2024

MengqingCao force-pushed the dvpp_support branch from 1386f6b to bc7b529 Compare January 19, 2024 03:37

Add more Ascend operators

866b914

1. add the Ascend dvpp calling code 2. add copyMakeBorder, cropResize and cropResizeMakeBorder 3. add resize with INTER_LINEAR and INTER_NEAREST interpolation

MengqingCao force-pushed the dvpp_support branch from bc7b529 to 866b914 Compare January 19, 2024 06:37

Merge branch '4.x' into dvpp_support

117964d

asmorkalov assigned fengyuentau and unassigned asmorkalov Mar 28, 2024

asmorkalov merged commit ab82106 into opencv:4.x Mar 28, 2024
10 checks passed

asmorkalov mentioned this pull request Apr 1, 2024

(5.x) Merge 4.x #3710

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add additional image processing operators for Ascend NPU by utilizing DVPP #3608

Add additional image processing operators for Ascend NPU by utilizing DVPP #3608

MengqingCao commented Dec 13, 2023

MengqingCao commented Jan 2, 2024 •

edited

Loading

opencv-alalek commented Jan 2, 2024

MengqingCao commented Jan 2, 2024

MengqingCao commented Jan 11, 2024

opencv-alalek commented Jan 11, 2024

opencv-alalek Jan 11, 2024

MengqingCao Jan 12, 2024

fengyuentau Jan 12, 2024

MengqingCao Jan 15, 2024 •

edited

Loading

fengyuentau left a comment

MengqingCao commented Jan 19, 2024

MengqingCao commented Jan 30, 2024

MengqingCao commented Feb 20, 2024

fengyuentau commented Feb 20, 2024

MengqingCao commented Feb 20, 2024

MengqingCao commented Feb 29, 2024

MengqingCao commented Mar 25, 2024 •

edited

Loading

Add additional image processing operators for Ascend NPU by utilizing DVPP #3608

Add additional image processing operators for Ascend NPU by utilizing DVPP #3608

Conversation

MengqingCao commented Dec 13, 2023

Pull Request Readiness Checklist

MengqingCao commented Jan 2, 2024 • edited Loading

opencv-alalek commented Jan 2, 2024

MengqingCao commented Jan 2, 2024

MengqingCao commented Jan 11, 2024

opencv-alalek commented Jan 11, 2024

opencv-alalek Jan 11, 2024

Choose a reason for hiding this comment

MengqingCao Jan 12, 2024

Choose a reason for hiding this comment

fengyuentau Jan 12, 2024

Choose a reason for hiding this comment

MengqingCao Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

fengyuentau left a comment

Choose a reason for hiding this comment

MengqingCao commented Jan 19, 2024

MengqingCao commented Jan 30, 2024

MengqingCao commented Feb 20, 2024

fengyuentau commented Feb 20, 2024

MengqingCao commented Feb 20, 2024

MengqingCao commented Feb 29, 2024

MengqingCao commented Mar 25, 2024 • edited Loading

MengqingCao commented Jan 2, 2024 •

edited

Loading

MengqingCao Jan 15, 2024 •

edited

Loading

MengqingCao commented Mar 25, 2024 •

edited

Loading