Reduce CUDA decoding latency by caching decoders across videos

[ahmadsharif1]

Currently CUDA decoding has a large latency overhead (almost 100ms) despite us caching the AVBufferRef* used for CUDA decoding:

torchcodec/src/torchcodec/decoders/_core/CudaDevice.cpp

Lines 136 to 139 in 4f3e491

	AVBufferRef* hw_device_ctx = getFromCache(device);
	if (hw_device_ctx != nullptr) {
	return hw_device_ctx;
	}

Repro instructions:

# 1. Create a conda env

conda create --name tcgpu1 --yes

conda activate tcgpu1



# 2. Install general pre-reqs

conda install conda libmamba --yes

$CONDA_PREFIX/bin/conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia --yes

$CONDA_PREFIX/bin/conda install nasm --yes

$CONDA_PREFIX/bin/conda install conda-forge::compilers --yes

$CONDA_PREFIX/bin/conda install pkg-config --yes

$CONDA_PREFIX/bin/conda install nvidia/label/cuda-12.4.0::cuda-toolkit --yes

$CONDA_PREFIX/bin/conda install nvidia/label/cuda-12.4.0::cuda-cudart --yes



# 3. Install nvidia video pre-reqs

TMPPATH=$(mktemp -d)

cd $TMPPATH

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git

cd nv-codec-headers

make PREFIX=$CONDA_PREFIX install





cd $TMPPATH

git clone https://github.com/FFmpeg/FFmpeg.git

cd FFmpeg



./configure --prefix=$CONDA_PREFIX --enable-nonfree --enable-cuda-nvcc --enable-libnpp --disable-static --enable-shared --optflags=-fno-omit-frame-pointer --disable-stripping

make -j install



# 4. Build and install torchcodec

cd $TMPPATH

git clone https://github.com/pytorch/torchcodec.git

cd torchcodec

git checkout origin/cuda_support

find . -name \*.so | xargs rm -rf; CMAKE_BUILD_PARALLEL_LEVEL=8 CXXFLAGS="" CMAKE_BUILD_TYPE=Release ENABLE_CUDA=1 ENABLE_NVTX=1 pip install -e . --no-build-isolation -vv --debug



# 5. Now you can run the GPU benchmark.

# Just for testing, run it on an existing small video that's part of the repo:

python benchmarks/decoders/gpu_benchmark.py --resize_devices=none



# 6. Now run it with the nsys profiler on a HD video.

nsys profile -t cuda,nvtx,osrt,nvvideo --gpu-video-device=all --event-sample system-wide --sampling-period=281250 -s system-wide python benchmarks/decoders/gpu_benchmark.py --resize_devices=none --devices=cuda:0 --video 853.mp4

This nsys profile will show a timeline where the decoder destruction takes a long time (see screenshot below).

In general it seems like when using FFMPEG either the GPU decoder creation or deletion takes a long time. This can be observed by carefully timing the decoder creation + decoding a single frame. (Note that timing this can be tricky because in general the first frame has higher overhead because it's a key frame).

In the code we are trying to cache the CUDA decoder by reusing the AVCodecContext::hw_device_ctx, but this is still slow because the destruction of the AVCodecContext takes a long time.

If I try to reuse the entire AVCodecContext, that requires a full flush of the buffers (otherwise the frame contents are corrupt because data is shared across videos) and that causes the subsequent call to nvdec_decoder_free() to take a long time (that is called by ff_hwaccel_uninit())