Add S12M_TIMECODE to av.sidedata.sidedata.Type Enum

[ivan94fi]

Overview

Adding S12M_TIMECODE field to the av.sidedata.sidedata.Type Enum would give the ability to read timecode information from SEI metadata.

Existing FFmpeg API

The existing FFmpeg enum is defined in libavutil/frame.h and is called AVFrameSideDataType.

The documentation is at https://ffmpeg.org/doxygen/trunk/group__lavu__frame.html#gae01fa7e427274293aacdf2adc17076bc.

The AV_FRAME_DATA_S12M_TIMECODE field is described as:

    /**
     * Timecode which conforms to SMPTE ST 12-1. The data is an array of 4 uint32_t
     * where the first uint32_t describes how many (1-3) of the other timecodes are used.
     * The timecode format is described in the documentation of av_timecode_get_smpte_from_framenum()
     * function in libavutil/timecode.h.
     */
    AV_FRAME_DATA_S12M_TIMECODE

This is the link to the AV_FRAME_DATA_S12M_TIMECODE field in the docs: https://ffmpeg.org/doxygen/trunk/group__lavu__frame.html#ggae01fa7e427274293aacdf2adc17076bca9f4e4ed5a874d1089ec07c384b81bb70

Expected PyAV API

PyAV should have a S12M_TIMECODE field in the av.sidedata.sidedata.Type Enum, which is currently defined as:

PyAV/av/sidedata/sidedata.pyx

Lines 11 to 29 in 1b27fc9

	Type = define_enum("Type", __name__, (
	("PANSCAN", lib.AV_FRAME_DATA_PANSCAN),
	("A53_CC", lib.AV_FRAME_DATA_A53_CC),
	("STEREO3D", lib.AV_FRAME_DATA_STEREO3D),
	("MATRIXENCODING", lib.AV_FRAME_DATA_MATRIXENCODING),
	("DOWNMIX_INFO", lib.AV_FRAME_DATA_DOWNMIX_INFO),
	("REPLAYGAIN", lib.AV_FRAME_DATA_REPLAYGAIN),
	("DISPLAYMATRIX", lib.AV_FRAME_DATA_DISPLAYMATRIX),
	("AFD", lib.AV_FRAME_DATA_AFD),
	("MOTION_VECTORS", lib.AV_FRAME_DATA_MOTION_VECTORS),
	("SKIP_SAMPLES", lib.AV_FRAME_DATA_SKIP_SAMPLES),
	("AUDIO_SERVICE_TYPE", lib.AV_FRAME_DATA_AUDIO_SERVICE_TYPE),
	("MASTERING_DISPLAY_METADATA", lib.AV_FRAME_DATA_MASTERING_DISPLAY_METADATA),
	("GOP_TIMECODE", lib.AV_FRAME_DATA_GOP_TIMECODE),
	("SPHERICAL", lib.AV_FRAME_DATA_SPHERICAL),
	("CONTENT_LIGHT_LEVEL", lib.AV_FRAME_DATA_CONTENT_LIGHT_LEVEL),
	("ICC_PROFILE", lib.AV_FRAME_DATA_ICC_PROFILE),
	("SEI_UNREGISTERED", lib.AV_FRAME_DATA_SEI_UNREGISTERED) if lib.AV_FRAME_DATA_SEI_UNREGISTERED != -1 else None,
	))

Example:

import av

container = av.open("video.mkv")

for frame in container.decode(video=0):
    timecode_data = frame.side_data.get(av.sidedata.sidedata.Type.S12M_TIMECODE)
    # ... use timecode_data

Investigation

N/A

Reproduction

N/A

Versions

• OS: Ubuntu 22.04
• PyAV runtime:
  This refers to the PyAV installation in a virtualenv, installed with pip install av. Not sure if it is useful:

PyAV v12.0.0
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-libtheora --disable-mediafoundation --disable-videotoolbox --enable-fontconfig --enable-gmp --enable-gnutls --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libxcb --enable-libxml2 --enable-lzma --enable-zlib --enable-version3 --enable-libx264 --disable-libopenh264 --enable-libx265 --enable-libxvid --enable-gpl
library license: GPL version 3 or later
libavcodec     60. 31.102
libavdevice    60.  3.100
libavfilter     9. 12.100
libavformat    60. 16.100
libavutil      58. 29.100
libswresample   4. 12.100
libswscale      7.  5.100

• PyAV build:
  N/A
• FFmpeg:
  This is the output of my system-wide ffmpeg installation, not sure it really matters here:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

Additional context

I tried manually adding a S12M_TIMECODE field to av.sidedata.sidedata, with this edits on the latest commit on main branch (1dfe776):

diff --git a/av/sidedata/sidedata.pyi b/av/sidedata/sidedata.pyi
index ac28f0d..e814bb2 100644
--- a/av/sidedata/sidedata.pyi
+++ b/av/sidedata/sidedata.pyi
@@ -23,6 +23,7 @@ class Type(EnumItem):
     CONTENT_LIGHT_LEVEL: int
     ICC_PROFILE: int
     SEI_UNREGISTERED: int
+    S12M_TIMECODE: int

 class SideData(Buffer):
     type: Type
diff --git a/av/sidedata/sidedata.pyx b/av/sidedata/sidedata.pyx
index 05ed002..1e62dff 100644
--- a/av/sidedata/sidedata.pyx
+++ b/av/sidedata/sidedata.pyx
@@ -26,6 +26,7 @@ Type = define_enum("Type", __name__, (
     ("CONTENT_LIGHT_LEVEL", lib.AV_FRAME_DATA_CONTENT_LIGHT_LEVEL),
     ("ICC_PROFILE", lib.AV_FRAME_DATA_ICC_PROFILE),
     ("SEI_UNREGISTERED", lib.AV_FRAME_DATA_SEI_UNREGISTERED) if lib.AV_FRAME_DATA_SEI_UNREGISTERED != -1 else None,
+    ("S12M_TIMECODE", lib.AV_FRAME_DATA_S12M_TIMECODE) if lib.AV_FRAME_DATA_S12M_TIMECODE != -1 else None,
 ))

But the compilation fails with this message:

Compiling av/sidedata/sidedata.pyx because it changed.
[1/1] Cythonizing av/sidedata/sidedata.pyx

Error compiling Cython file:
------------------------------------------------------------
...
    ("GOP_TIMECODE", lib.AV_FRAME_DATA_GOP_TIMECODE),
    ("SPHERICAL", lib.AV_FRAME_DATA_SPHERICAL),
    ("CONTENT_LIGHT_LEVEL", lib.AV_FRAME_DATA_CONTENT_LIGHT_LEVEL),
    ("ICC_PROFILE", lib.AV_FRAME_DATA_ICC_PROFILE),
    ("SEI_UNREGISTERED", lib.AV_FRAME_DATA_SEI_UNREGISTERED) if lib.AV_FRAME_DATA_SEI_UNREGISTERED != -1 else None,
    ("S12M_TIMECODE", lib.AV_FRAME_DATA_S12M_TIMECODE) if lib.AV_FRAME_DATA_S12M_TIMECODE != -1 else None,
                                                             ^
------------------------------------------------------------

av/sidedata/sidedata.pyx:29:61: cimported module has no attribute 'AV_FRAME_DATA_S12M_TIMECODE'

Error compiling Cython file:
------------------------------------------------------------
...
    ("GOP_TIMECODE", lib.AV_FRAME_DATA_GOP_TIMECODE),
    ("SPHERICAL", lib.AV_FRAME_DATA_SPHERICAL),
    ("CONTENT_LIGHT_LEVEL", lib.AV_FRAME_DATA_CONTENT_LIGHT_LEVEL),
    ("ICC_PROFILE", lib.AV_FRAME_DATA_ICC_PROFILE),
    ("SEI_UNREGISTERED", lib.AV_FRAME_DATA_SEI_UNREGISTERED) if lib.AV_FRAME_DATA_SEI_UNREGISTERED != -1 else None,
    ("S12M_TIMECODE", lib.AV_FRAME_DATA_S12M_TIMECODE) if lib.AV_FRAME_DATA_S12M_TIMECODE != -1 else None,
                         ^
------------------------------------------------------------

av/sidedata/sidedata.pyx:29:25: cimported module has no attribute 'AV_FRAME_DATA_S12M_TIMECODE'
Traceback (most recent call last):
  File "/code/PyAV/setup.py", line 156, in <module>
    ext_modules += cythonize(
  File "/code/PyAV/venvs/Linux.6.5.0-21-generic.cpython3.10/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/code/PyAV/venvs/Linux.6.5.0-21-generic.cpython3.10/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: av/sidedata/sidedata.pyx
make: *** [Makefile:14: build] Error 1

From FFmpeg sources, I can see that AV_FRAME_DATA_S12M_TIMECODE was added in libavutil version 56.20.100:

https://github.com/FFmpeg/FFmpeg/blob/89215237dd6ac64f94e14aa20a000e0440a00aac/doc/APIchanges#L894-L895

And I have installed the package libavutil-dev version 7:4.4.2-0ubuntu0.22.04.1 and libavutil56, which for Ubuntu22.04 provide libavutil version 56.70.100 as can be seen in the following lines:

$ dpkg -L libavutil-dev
# ....
/usr/lib/x86_64-linux-gnu/pkgconfig/libavutil.pc
# ....
$ cat /usr/lib/x86_64-linux-gnu/pkgconfig/libavutil.pc
prefix=/usr
exec_prefix=${prefix}
libdir=/usr/lib/x86_64-linux-gnu
includedir=/usr/include/x86_64-linux-gnu

Name: libavutil
Description: FFmpeg utility library
Version: 56.70.100
Requires:
Requires.private:
Conflicts:
Libs: -L${libdir}  -lavutil
Libs.private: -pthread -lva-drm -lva -lva-x11 -lva -lvdpau -lX11 -lm -ldrm -lmfx -lstdc++ -ldl -lOpenCL -lva -lXv -lX11 -lXext
Cflags: -I${includedir}

$ dpkg -L libavutil56
# ...
/usr/lib/x86_64-linux-gnu/libavutil.so.56.70.100
# ...

So the libavutil version should be sufficient, as far as I understand, but maybe I need something else to make this work, I do not have sufficient experience.