Any chance we can load the onnx model and run it into TensorRT

[nelsontseng0704]

Such a great work. I am looking for quicker inference for GFPGAN. Is there any chance we can load the GFPGAN.onnx and run it on TensorRT? Looking forward to your suggestion.

Thanks.

[lschaupp]

Yes I made it work. I am getting 70 ms runtime on 3060 RTX for the 512 upscale model. (GFPGAN 1.4) using TensorRT. (without pre and post processing)
However the ONNX model itself has some quality decrease in comparison to the original model for some reason.

[nelsontseng0704]

Thanks for the sharing. I follow your fork from Face-Restoration-TensorRT and I tried to build the same fgpgan tensorrt, however I encounter the following error, do I miss something on the CMakeLists.txt?

[ 20%] Building CXX object CMakeFiles/demo.dir/demo.cpp.o In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:26, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h: In member function ‘virtual bool nvinfer1::IGpuAllocator::deallocate(void*)’: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h:479:26: warning: ‘virtual void nvinfer1::IGpuAllocator::free(void*)’ is deprecated [-Wdeprecated-declarations] 479 | this->free(memory); | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h:410:33: note: declared here 410 | TRT_DEPRECATED virtual void free(void* const memory) noexcept = 0; | ^~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h: At global scope: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:866:104: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 866 | virtual IPluginV2* createPlugin(AsciiChar const* name, PluginFieldCollection const* fc) noexcept = 0; | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:878:11: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 878 | = 0; | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntime.h:22, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:17, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferImpl.h:648:47: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 648 | virtual IPluginV2& getPlugin() noexcept = 0; | ^ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntime.h:22, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:17, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferImpl.h:1014:113: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 1014 | virtual IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept = 0; | ^ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:3647:28: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 3647 | IPluginV2& getPlugin() noexcept | ^~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:6965:88: warning: ‘IFullyConnectedLayer’ is deprecated [-Wdeprecated-declarations] 6965 | ITensor& input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights) noexcept | ^~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:1518:22: note: declared here 1518 | class TRT_DEPRECATED IFullyConnectedLayer : public ILayer | ^~~~~~~~~~~~~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:7542:101: warning: ‘IRNNv2Layer’ is deprecated [-Wdeprecated-declarations] 7542 | ITensor& input, int32_t layerCount, int32_t hiddenSize, int32_t maxSeqLen, RNNOperation op) noexcept | ^~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:3349:22: note: declared here 3349 | class TRT_DEPRECATED IRNNv2Layer : public ILayer | ^~~~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:7617:94: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 7617 | IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept | ^~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /usr/include/c++/9/ratio:38, from /usr/include/c++/9/chrono:38, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:1: /usr/include/c++/9/type_traits: In instantiation of ‘struct std::is_polymorphic<FaceRestore>’: /home/paperspace/.local/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h:1516:30: required from ‘class pybind11::class_<FaceRestore>’ /home/paperspace/Face-Restoration-TensorRT/demo.cpp:24:45: required from here /usr/include/c++/9/type_traits:709:12: error: invalid use of incomplete type ‘class FaceRestore’ 709 | struct is_polymorphic | ^~~~~~~~~~~~~~ compilation terminated due to -Wfatal-errors. make[2]: *** [CMakeFiles/demo.dir/build.make:76: CMakeFiles/demo.dir/demo.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/demo.dir/all] Error 2 make: *** [Makefile:91: all] Error 2

[lschaupp]

@nelsontseng0704 the fork has a different purpose. It is intended to construct a python wrapper around the C++ code for inference. For GFPGAN I made it work with the official repository.

Also it is heavily WIP. So no wonder you are getting some errors when trying to build it :)

[JieLuChen]

Yes I made it work. I am getting 70 ms runtime on 3060 RTX for the 512 upscale model. (GFPGAN 1.4) using TensorRT. (without pre and post processing) However the ONNX model itself has some quality decrease in comparison to the original model for some reason.

Got it working with model.onnx (including inference)
also successfully converted model512.onnx to model2.engine
But when running ./demo ../models/model2.engine -i ../images/test.png
I always get Cuda failure: 700
Aborted (core dumped)

Do you know what the fix is for this?

[lschaupp]

You would need to debug the error within the C++ code. Cuda 700 is a standard error which does not tell much.