HEVC/h265 support for GStreamer on Raspbery Pi 4B/400/5

[fduncanh]

Now that UxPlay 1.70 supports HEVC/h265/4K video, this issue thread is for discussion of how to get the stateless Video4Linux2 HEVC driver(s) on Raspberry Pi 4B/400/5 working with the stateless V4l2 plugin in gstreamer1.0-plugins-bad

• this continues discussion in Trying to add h265 support (for modern clients with wired connection) (was *** ERROR: unsupported h265 video detected) #233

LibreELEC/Kodi have HEVC on these Pi's working using a modified ffmpeg created by R Pi engineers:
see jc-kynesim/rpi-ffmpeg#94 (comment)

and https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

• The gstreamer code is at https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs?ref_type=heads

• see here for useful info from LibreELEC
  https://wiki.libreelec.tv/hardware/raspberry-pi

• if we can understand how to access the HEVC drivers on these R Pi models working with UxPlay , perhaps with a patch to gstreamer-plugins-bad, a pull request to gstreamer can be made (this was done in the past to get GPU Hardware H264 decoding on R Pi 4B and earlier working with UxPlay (with the stateful v4l2 plugin in gstreamer-plugins-good).

• The stateless V$L2 API is described below: hopefully this is already mainly implemented by gstreamer
  https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/dev-stateless-decoder.html

[connorh315]

Just as a minor bit of information. I've nearly gotten ffmpeg working with uxplay - Currently the frames are all green for some reason but I'll iron that out soon. Main thing is though that the decoder block on the RPi 4 is capable of decoding the AirPlay H265 packets and it seems to run at a relatively good frame rate (hard to tell as the stream looks shocking). The relevant changes just needs to be pushed into gstreamer now.

[fduncanh]

I have some RPI 5's and can dedicate one to testing too.

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

Stateless v4l2 justs queues an encoded frame in the "OUTPUT" (from encoder) queue
and dequeues if from the "CAPTURE" queue, so interaction with the driver which connects to the hardware is
relatively simple. I just dont yet know how the interface to the driver works, and where it is in the ffmpeg code.

[fduncanh]

more info from jc-kinesim

The driver is in the PiOS linux tree - you can find it under driver/staging/media/rpivid/*. Hevc decode is a completely independent h/w block - GPU not involved. I'm sure the h/w is proprietary, I'm not completely sure to whom, so you are on your own for figuring out what is going on there (but honestly the h/w setup corresponds quite closely to the info passed in the V4L2 stateless structures so most of it is quite obvious).

[fduncanh]

current R Pi OS HEVC driver

https://github.com/raspberrypi/linux/tree/rpi-6.6.y/drivers/staging/media/rpivid

[connorh315]

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

RPi-FFmpeg is quite easy to build, unfortunately it just takes quite a while (~30 minutes on a RPi 4 with a decent SD card). I expect that subsequent rebuilds with minor changes would be faster but I haven't tried. Just something worth mentioning is that the hardware decoder outputs a DRM-frame which doesn't seem to be supported by many graphics libraries. I originally used SDL and tried to copy the hardware frame via the CPU into main memory to then be converted to YUV and eventually output back onto the display however it seemed to make those green frames that I was talking about earlier. Instead I'm using a OpenGL instance now, as I'm relatively sure that it can directly render DRM-Prime frames without any conversion/copying. I just want to get a working prototype at the moment and test that it can decode relatively well (it seemed relatively stable with an iPhone 15 Pro, it was decent with an iPad pro and i'll test my macbook later once I've actually got it stably outputting frames (can force the full 3840x2160 output with that) to stress test it.

[connorh315]

Might have to give up with trying to get ffmpeg imported. Non-stop getting green-tinted frames from the decode and I can't seem to figure out what I'm doing wrong. Including ffmpeg into uxplay is relatively easy though (including adding hw-accelerated). Very frustrating

[fduncanh]

I am trying to find out why gstreamer is not detecting the rpivid driver on a R Pi 5.

The rpivid_hevc driver is loaded (see it in lsmod), the gstreamer libv4l2codec is correctly compiled and present (installed by gstreamer-1.0-plugins-bad)

"gstreamer-inspect-1.0 v4l2codec" shows the drivers, but lists "0 features" which means that no v4l2 devices was detected.

however, the v4l2 devices can be listed with "v4l2-ctl --list-devices" (v4l2-ctl is in the "v4l-utils" package, which is probably installed, if not, install with apt on R Pi OS)

This shows that the rpivid kernel model (on R Pi OS) is connected to /dev/video19 on my system. (and /dev/media0)

"v4l2-ctl -d19 -D" lists all its details, including that it is enabled.

[fduncanh]

posted a query about this on GStreamer discourse board: deleted it

$v4l2-ctl --list-devices
<snip>
rpivid (platform:rpivid):
        /dev/video19
        /dev/media0

$v4l2-ctl -d19 -D
Driver Info:
        Driver name      : rpivid
        Card type        : rpivid
        Bus info         : platform:rpivid
        Driver version   : 6.6.51
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : rpivid
        Model            : rpivid
        Serial           : 
        Bus info         : platform:rpivid
        Media version    : 6.6.51
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.6.51
Interface Info:
        ID               : 0x0300000c
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : rpivid-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000004 of entity 'rpivid-proc' (Video Decoder): Data, Enabled, Immutable

[fduncanh]

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

[connorh315]

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

I was reading somewhere that the output of the decoder is SAND8/SAND30... Maybe that's where the issue lies?

[fduncanh]

The problem is much more basic than that:

   src_caps = gst_v4l2_decoder_enum_src_formats (decoder);

  if (gst_caps_is_empty (src_caps)) {
    GST_WARNING ("Not registering H265 decoder since it produces no "
        "supported format");
    goto done;

• the ioctl calls to VIDIOC_G_FMT and VIDIOC_ENUM_FMT (to interrogate the decoder) dont fail, but the caps that gst_v4l2_decoder_enum_src_formats(decoder) extracts from the results and then returns are EMPTY when converted to a string with gst_caps_to_string(caps) This causes rpivid to be rejected as a v4l2 decoder.

• I need to find out what is in the responses to the two v4l2 calls. (which means learning about them)

• It looks like they are being analyzed for things the driver cant yet know about (pixelformat, width, height), maybe its an error for this code to be used as a registration test when gst-inspect-1.0 is run, ???? (i'll try commenting out the test)

      gst_v4l2_decoder_probe_caps_for_format (self,
      fmt.fmt.pix_mp.pixelformat, fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);

GstCaps *
gst_v4l2_decoder_enum_src_formats (GstV4l2Decoder * self)
{
  gint ret;
  struct v4l2_format fmt = {
    .type = self->src_buf_type,
  };
  GstCaps *caps;
  gint i;

  g_return_val_if_fail (self->opened, FALSE);

  ret = ioctl (self->video_fd, VIDIOC_G_FMT, &fmt);
  if (ret < 0) {
    GST_ERROR_OBJECT (self, "VIDIOC_G_FMT failed: %s", g_strerror (errno));
    return FALSE;
  }

  caps =
      gst_v4l2_decoder_probe_caps_for_format (self,
      fmt.fmt.pix_mp.pixelformat, fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);

  /* And then enumerate other possible formats and place that as a second
   * structure in the caps */
  for (i = 0; ret >= 0; i++) {
    struct v4l2_fmtdesc fmtdesc = { i, self->src_buf_type, };
    GstCaps *tmp;

    ret = ioctl (self->video_fd, VIDIOC_ENUM_FMT, &fmtdesc);
    if (ret < 0) {
      if (errno != EINVAL)
        GST_ERROR_OBJECT (self, "VIDIOC_ENUM_FMT failed: %s",
            g_strerror (errno));
      continue;
    }

    tmp = gst_v4l2_decoder_probe_caps_for_format (self, fmtdesc.pixelformat,
        fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);
    caps = gst_caps_merge (caps, tmp);
  }

  return caps;
}

[fduncanh]

HOORAY!!!!

commenting out that test indeed allows the "v4l2slh265dec V4L2 Stateless H.265 Video Decoder" to be registered, and show up with "1 feature" in "gst-inspect-1.0 v4l2codec"

[connorh315]

And it works???

[fduncanh]

not yet, sorry
GST_DEBUG=2
The caps are returned as "empty"

v4l2slh265dec0 error Unsupported bitdepth/chroma format
v4l2slh265dec0 error No support for 2896x2160 8bit chroma IDC1
v4l2slh265dec0 Failed to negotiate with downstream
v4l2slh265dec0 subclass does not want accept new sequence
v4l2slh265dec0 Failed to process sps
queue error internal data stream error
queue error streaming stopped, reason not-negotiated (-4)

but it's a start......

[fduncanh]

(without UxPlay, just run gst-inspect-1.0)

The rpivid driver is reporting valid formats in response to VIDIOC_G_FMT and VIDIOC_ENUM_FMT ioctl calls

VIDIOC_G_FMT:
width 1920
height 1088
pixelformat   (fourcc code)   32 31 43 4e    NC12

VIDIOC_ENUM_FMT:

index   type   flags    pixelformat      description
0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"
1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"

• gst_v4l2_decoder_probe_caps_for_format is not converting them into non-empty caps.

[fduncanh]

static GstCaps *
gst_v4l2_decoder_probe_caps_for_format (GstV4l2Decoder * self,
    guint32 pixelformat, gint unscaled_width, gint unscaled_height)
{
  gint index = 0;
  GstCaps *caps, *tmp, *size_caps;
  GstVideoFormat format;
  guint32 drm_fourcc;

  GST_DEBUG_OBJECT (self, "enumerate size for %" GST_FOURCC_FORMAT,
      GST_FOURCC_ARGS (pixelformat));

  if (!gst_v4l2_format_to_video_format (pixelformat, &format))    <+++++++++++++++++++++this is what fails!!!
    return gst_caps_new_empty ();

  caps = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
      gst_video_format_to_string (format), NULL);

  size_caps = gst_caps_new_empty ();
  while ((tmp = gst_v4l2_decoder_enum_size_for_format (self, pixelformat,
              index++, unscaled_width, unscaled_height))) {
    size_caps = gst_caps_merge (size_caps, tmp);
  }

  if (!gst_caps_is_empty (size_caps)) {
    tmp = caps;
    caps = gst_caps_intersect_full (tmp, size_caps, GST_CAPS_INTERSECT_FIRST);
    gst_caps_unref (tmp);
  }

  /* TODO: Add a V4L2 to DRM fourcc translator for formats that we don't support
   * in software.
   */
  drm_fourcc = gst_video_dma_drm_fourcc_from_format (format);
  if (drm_fourcc /* != DRM_FORMAT_INVALID */ ) {
    GstCaps *drm_caps;

    drm_caps = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
        "DMA_DRM", "drm-format", G_TYPE_STRING,
        gst_video_dma_drm_fourcc_to_string (drm_fourcc, 0), NULL);
    gst_caps_set_features_simple (drm_caps,
        gst_caps_features_new_single_static_str
        (GST_CAPS_FEATURE_MEMORY_DMABUF));

    if (!gst_caps_is_empty (size_caps)) {
      gst_caps_set_features_simple (size_caps,
          gst_caps_features_new_single_static_str
          (GST_CAPS_FEATURE_MEMORY_DMABUF));
      tmp = drm_caps;
      drm_caps =
          gst_caps_intersect_full (tmp, size_caps, GST_CAPS_INTERSECT_FIRST);
      gst_caps_unref (tmp);
    }

    caps = gst_caps_merge (drm_caps, caps);
  }

  gst_caps_unref (size_caps);

  return caps;
}

gboolean
gst_v4l2_format_to_video_format (guint32 pix_fmt, GstVideoFormat * out_format)
{
  struct FormatEntry *entry = lookup_v4l2_fmt (pix_fmt);

  if (!entry)
    return FALSE;

  *out_format = entry->gst_fmt;
  return TRUE;
}

• I think the issue is that pixelformats NC12, NC30 are not in this table!

static struct FormatEntry format_map[] = {
  {V4L2_PIX_FMT_NV12, 1, GST_VIDEO_FORMAT_NV12, 8, 420},
  {V4L2_PIX_FMT_YUYV, 1, GST_VIDEO_FORMAT_YUY2, 8, 422},
  {V4L2_PIX_FMT_SUNXI_TILED_NV12, 1, GST_VIDEO_FORMAT_NV12_32L32, 8, 422},
  {V4L2_PIX_FMT_NV12_4L4, 1, GST_VIDEO_FORMAT_NV12_4L4, 8, 420},
  {V4L2_PIX_FMT_MM21, 2, GST_VIDEO_FORMAT_NV12_16L32S, 8, 420},
  {V4L2_PIX_FMT_YUV420M, 3, GST_VIDEO_FORMAT_I420, 8, 420},
  {V4L2_PIX_FMT_P010, 1, GST_VIDEO_FORMAT_P010_10LE, 16, 420},
  {V4L2_PIX_FMT_NV15_4L4, 1, GST_VIDEO_FORMAT_NV12_10LE40_4L4, 10, 420},
  {V4L2_PIX_FMT_MT2110T, 2, GST_VIDEO_FORMAT_MT2110T, 10, 420},
  {V4L2_PIX_FMT_MT2110R, 2, GST_VIDEO_FORMAT_MT2110R, 10, 420},
  {0,}
};

static struct FormatEntry *
lookup_v4l2_fmt (guint v4l2_pix_fmt)
{
  gint i;
  struct FormatEntry *ret = NULL;

  for (i = 0; format_map[i].v4l2_pix_fmt; i++) {
    if (format_map[i].v4l2_pix_fmt == v4l2_pix_fmt) {
      ret = format_map + i;
      break;
    }
  }

  return ret;
}

[fduncanh]

• so the issue is the unusual fourcc pixelformat codes NC12 and NC30. I havent yet mange to find them listed anywhere.

VIDIOC_G_FMT:
width 1920
height 1088
pixelformat   (fourcc code)   32 31 43 4e    NC12

VIDIOC_ENUM_FMT:

index   type   flags    pixelformat      description
0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"
1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"

[connorh315]

• so the issue is the unusual fourcc pixelformat codes NC12 and NC30. I havent yet mange to find them listed anywhere.

VIDIOC_G_FMT:

width 1920

height 1088

pixelformat   (fourcc code)   32 31 43 4e    NC12



VIDIOC_ENUM_FMT:



index   type   flags    pixelformat      description

0           9     0     32 31 43 4e  NC12   "Y/CbCr 4:2:0 (120b cols)"

1           9     0     32 31 43 4e  NC30   "10-bit Y/CbCr 4:2:0 (120b cols)"

Those are the SAND formats I was referencing earlier. (Post 12 of https://forums.raspberrypi.com/viewtopic.php?t=343593)

[fduncanh]

OK! I'm guessing "sand" refers to the proprietary hevc decoding block hardware in R PI.

Well I got to understand enough to discover that by myself!

The "unsand" code in jc-kinesim's ffmped rpi-ffmpeg work is

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavfilter/vf_unsand.c

[fduncanh]

old code (2008)

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavfilter/vf_unsand.c

new SAND code. from R Pi (committed by jc-kinesim dec 2023)jc-kynesim/rpi-ffmpeg@b6b137b

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fn_pw.h

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fns.c

https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/6.0.1/main/libavutil/rpi_sand_fns.h

[fduncanh]

I have asked jc-kinesim whether this R Pi Foundation code (copyright date 2018) replaces or works with the older non-RPi vf_unsand.c jc-kynesim/rpi-ffmpeg#94 (comment)

Its a LOT of code. Maybe too big to be accepted into gstreamer? I wonder if it could be accessed by checking whether libav locally contains this code or not.

this is the commit in dec 2023 at rpi-ffmpeg.

jc-kynesim/rpi-ffmpeg@b6b137b

[fduncanh]

Its possible that UxPlay could offer a gstreamer patch that UxPlay users could use to rebuild libgstv4l2codecs.c

This is how it supported gstreamer's v4l2 (stateful h264 decode) before I got fixes accepted into gstreamer.

[fduncanh]

dependecies of RPI sand code

#include "libavutil/frame.h" this might drag in a lot, should check if gstreamer-libav uses it.
#include "avassert.h" (presumably replaceable by assert.h)
#include "frame.h" same as libavutil/frame.h ???

[connorh315]

I doubt that you would want this as it's just adding unnecessary bloat on the project, however I've just gotten FFmpeg working in UxPlay and correctly decoding H.265 streams. Worst comes to worst that can be an alternative provision?

[fduncanh]

well now at least we know what the issues are.

$v4l2-ctl --list-formats-ext -d 19
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NC12' (Y/CbCr 4:2:0 (128b cols))
        [1]: 'NC30' (10-bit Y/CbCr 4:2:0 (128b cols))

SAND pixelformats fourcc = NC12, NC30 are not supported in gstreamer-plugins-bad v4l2codecs

[fduncanh]

vf_unsand .c was also added Dec 2023 its much older, Dec 2023 is just the date when kernel 6.0.1 was added

jc-kynesim/rpi-ffmpeg@c7d8474

[fduncanh]

some more into

A response from @6by9 (R Pi engineer) to my post at jc-kynesim/rpi-ffmpeg#94 (comment)

AIUI Mainline GStreamer will only support formats that are defined in the mainline kernel, and upstreaming the HEVC decoder is still in progress (I need to talk to jc-kynesim about it this coming week).

Once the format and driver are in the mainline kernel, I wouldn't expect there to be any great obstacles to adding the format to GStreamer, but would ask Nicolas Dufresne as the person who knows that area best.

Sand was an internal Broadcom name for the format as it originally came from a company called Sand Video that was bought by them in 2004 - https://www.theregister.com/2004/04/13/broadcom_acquires_sand/

I also posted on Gstreamer forum:

https://discourse.gstreamer.org/t/v4l2codecs-feasibility-of-adding-support-for-raspberry-pi-hevc-sand-formats-nc12-nc30/3522

[fduncanh]

some more activity:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7355

maybe some action at R Pi might get stirred up by my queries.

[fduncanh]

@connorh315

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7355
there is interesting R Pi HEVC actvity at the above link today

[fduncanh]

@connorh315 more news, the R Pi HEVC driver has at last been submitted to the Linux kernel for review.....

https://lore.kernel.org/lkml/[email protected]/