Incorrect colours on Raspberry Pi 4, -avdec is faster than hardware decoding

[connorh315]

Receiver device running UxPlay: Raspberry Pi 4
Sender: iPad Pro (M1) | iPhone 15 Pro
OS Version: Most recent Raspberry Pi OS (64-bit) with Desktop - "Debian GNU/Linux 12 (bookworm)" - released 2024-07-04
GStreamer version - 1.22.0
UxPlay version: 1.69
Launch command: ./uxplay -p -d
Videosink: any

My Raspberry Pi 4 is not showing certain colours from the Airplay stream. It is the exact same as the issue described here: #313. The only difference is that I know it cannot be the sender device not sending the correct colours as the stream looks perfect on my Windows machine running UxPlay. As such I can only assume that it is a Linux GStreamer issue. I have tried xvimagesink, ximagesink and glimagesink and they don't make a difference. Just as some additional information, -avdec produces a smoother stream than hardware decoding on all the videosinks which is odd. I have reinstalled the Raspberry Pi OS twice now to check and used on two different Raspberry Pi 4 boards

This is not an issue with the sender device sending poor streams, as my UxPlay instance running on Windows does not have the same colour issues.

[fduncanh]

I see it.

seems OK on linux, but the red seems brown and the blue is less bright on r pi.

https://www.youtube.com/watch?v=lnrJASmgHBM

[connorh315]

There seems to be a big issue with greys as well, some greys just do not show up at all and instead are rendered as white only. I'll try and grab a screenshot in a bit.

[fduncanh]

I compared the above image on R Pi with (side by side) (1) the image shown on firefox on R Pi.
(2) the image on uxplay on R Pi, transmitted from the screen if an iPad showing it.

on RPi firefox and UxPlay seem to have identical colors so it seems to be a R Pi issue, not UxPlay ?
please do this test for yourself and report what you see.

[fduncanh]

please find a link to a greyscale image on web, that the viewing-side-by-side-on-browser-and-uxplay test can be done on.

[fduncanh]

is this a fix?

https://forums.raspberrypi.com/viewtopic.php?t=304842

EDIT: I did this but am not sure if it changes anything?

[connorh315]

I tried that one already and I don't think it made any improvements either :(

[fduncanh]

In any case since uxplay and firefox show identical colors (do you agree?) , its not a uxplay or gstreamer problem.

There should be a fix to be found out there on the web?

[connorh315]

Definitely, will close this - Once I've found a solution I'll post it here.

[connorh315]

Sorry, might've prematurely closed this, please see attached image; It shows a screenshot of the iPad screen, compared against the actual UxPlay stream:

--Photo removed--

You can see the difference is mainly with greys being off, or just converted to white?

[connorh315]

Attached here is the UxPlay stream as seen from a Windows PC, showing that the correct colours are being sent by the sender:

--Photo removed--

[fduncanh]

I see the grey issue. so that is uxplay/gstreamer.

what about the color issue?

[connorh315]

Sorry I hope that I didn't come across like it was a vibrant colour issue, I meant that the grey "colour" was showing up as white. I probably should've put a reference image there 😬

[fduncanh]

Can you find a link to or post an appropriate test image for the grey issue? thanks.

(Something that is easy to display on the iPad screen with a standard app, preferably a browser.)

[connorh315]

If you go to this website, the hero banner at the top has the colours #ffffff and #f5f5f5 right in the middle and the two colours show up as white despite one being a grey. Additionally, I have noticed that the colours on this site too are all displaying as white, despite the fact that some of them are actually a light-green or a light-blue

[connorh315]

Issue also exists on Ubuntu Desktop for Raspberry Pi 4, running GStreamer version 1.24.x (can't remember minor revision)

[connorh315]

Just for reference, using -vdmp and dumping the h264 stream, I can replay it in VLC on the same raspberry pi and the grey colour shows fine.

[fduncanh]

@connorh315 great investigating! this is exactly the type of info needed!

for uxplay, we need to know if the problem is the same with -avdec and -v4l2.
as these are two different image rendering engines (-avdec in software, -v4l2 in RPI4 hardware)

[connorh315]

@fduncanh issue occurs with both -v4l2 and -avdec. For reference as well, I've also tried replaying the h264 dump with this command just here:
gst-launch-1.0 filesrc location=videodump.h264 ! h264parse ! decodebin ! xvimagesink. This pipeline is as simple as it gets and yet still the same issue occurs. Changing the videosink to glimagesink has no impact either.

[fduncanh]

Does the videodump.h264 look OK with e.g vlc?

what about with Gstreamer on Linux on a desktop machine x86_64 etc.

I assume its R Pi specific?

is the videodump file able to be made public?

That way one can post an issue to GStreamer developers.

[connorh315]

Does the videodump.h264 look OK with e.g vlc?

Yes looks absolutely fine when played back with VLC

what about with Gstreamer on Linux on a desktop machine x86_64 etc.

Actually the same issue exists on a x86_64 machine! The machine I've just tried on is an Ubuntu 24.04.1 install running on a Lenovo laptop. It has GStreamer 1.24.2 as default.

I assume its R Pi specific?

I think so. Again will test in a second. As just discovered, same issues exists on the Ubuntu machine.

is the videodump file able to be made public?

Yes: videodump.h264.zip (It had to be .zip, I couldn't upload a raw .h264 file as it wasn't in GitHub's whitelist)

[PancakeTAS]

I believe AirPlay runs in full color space (0-255) rather than limited (16-235) but this is simply ignored when playing back, which is why the top 20 and bottom 16 colors are clamped to either black or white. This issue still persists in the current release and the -bt709 flag doesn't seem to do anything either. Somewhere in the code the color range needs to be updated and I have no idea where :P

[fduncanh]

This is a GStreamer issue. I thought I had fixed it by a PR accepted into GStreamer-1.20.4 but i seems the need from -bt709 opt cam back in GStreamer-1.22.0.

@connorh315 's posts above show it is a GStreamer issue (using a uxplay raw video dump).

Prior to my accepted path GStreamer's BT709 ( colorspace only recognized limited color spaces by airplay uses 1:3:5:1

This is my PR in GStreamer, which is only in Video4Linux2 code (for the stateful driver for the Broadcom GPU h264 decoder)

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/commit/197b72f32afa245f472a017fd07f82aefe4ea7c2

The fact that @connorh315 suggests it is also an issue with -avdec (software decoder) suggests that there are other parts of GStreamer involving BT709 that need a fix.

Your statement about top 20 and bottom 16 colors being clamped to white is very useful.

How did you see this?

[fduncanh]

Can this be demonstrated on a Desktop system (not RPi) or a RPi model 5? Or just on R Pi 4B or earlier?
(We also have R Pi 5 available.)

Can you provide the test explicitly showing the missing colors?

[fduncanh]

@connorh315 mentioned

the only difference is that I know it cannot be the sender device not sending the correct colours as the stream looks perfect on my Windows machine running UxPlay. As such I can only assume that it is a Linux GStreamer issue. I have tried xvimagesink, ximagesink and glimagesink and they don't make a difference.

The widows videosink is d3d11videosink, seems not to have the issue.

[fduncanh]

is it possible this is a monitor setting issue? EDIT: NO IT'S NOT
https://linustechtips.com/topic/1215945-full-rbg-or-limited-rgb-for-pc-monitor/

[fduncanh]

https://g5-staging64.benq.eu/en-is/knowledge-center/knowledge/full-rgb-vs-limited-rgb-is-there-a-difference.html

[PancakeTAS]

How did you see this?

It's really obvious on my monitor, but this photo here helps as well. If top and bottom look the same, then the color is clamped

[fduncanh]

@PancakeTAS I the black on the right is the same (top and bottom), but yes, UxPlay mirror mode on Linux sees both top and bottom left as white.

Same as https://www.schemecolor.com/grey-and-white-color-scheme.php example from @connorh315

[fduncanh]

@PancakeTAS I the black on the right is the same (top and bottom), but yes, UxPlay mirror mode on Linux sees both top and bottom left as white.

Same as https://www.schemecolor.com/grey-and-white-color-scheme.php example from @connorh315

I posted about this at GStreamer
https://discourse.gstreamer.org/t/gstreamer-apparently-rendering-full-range-color-0-255-as-limited-range-16-235-is-this-a-gstreamer-or-monitor-issue/3782

[fduncanh]

We are back to this BT709 "legal" vs "Full range" issue. Will try to collect refs here.

https://community.avid.com/forums/t/202331.aspx

I speculate that the issues in xvimagesink are somewhere in here:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-base/sys/xvimage/xvcontext.c?ref_type=heads

[PancakeTAS]

I "fixed" the issue with this change here: 5a64fcc.
I've never worked with gstreamer before, but I assume the videoconvert in there will simply convert the full range back to limited range. It works but isn't a proper solution.

[fduncanh]

@PancakeTAS

good work!.

I also got a reply on:
https://discourse.gstreamer.org/t/gstreamer-apparently-rendering-full-range-color-0-255-as-limited-range-16-235-is-this-a-gstreamer-or-monitor-issue/3782

The issue is also fixed for glimagesink (but not yet for xvimagesink) by improvements in the upcoming GStreamer-1.26.x series. I tested on a build of the development branch GStreamer 1.25.0.1 and verified it.

glcolorconvert: add support for planar yuv-> planar yuv conversion (!6596) · Merge requests · GStreamer / gstreamer · GitLab 1 is the glcolorconvert(glimagesink) change to support arbitrary color matrices which will be part of the 1.26 release series.

[fduncanh]

@PancakeTAS interestingly GStreamer is now showing colorimetry= 1:3:7:1 instead of 1:3:5;1 which what it was showing some years ago in GStreamer-1.18.6 on Raspberry Pi. (which also now shows 1:3:7:1)

/* apple uses colorimetry=1:3:5:1                                * //now seems to be 1:3:7:1
 * (not recognized by v4l2 plugin in Gstreamer  < 1.20.4)        *
 * See .../gst-libs/gst/video/video-color.h in gst-plugins-base  *
 * range = 1   -> GST_VIDEO_COLOR_RANGE_0_255      ("full RGB")  * 
 * matrix = 3  -> GST_VIDEO_COLOR_MATRIX_BT709                   *
 * transfer = 5 -> GST_VIDEO_TRANSFER_BT709                      *  // now seems to be 7 -> VIDEO_TRANSFER_SRGB
 * primaries = 1 -> GST_VIDEO_COLOR_PRIMARIES_BT709              *
 * closest used by  GStreamer < 1.20.4 is BT709, 2:3:5:1 with    *                            *
 * range = 2 -> GST_VIDEO_COLOR_RANGE_16_235 ("limited RGB")     */  

static const GstVideoColorimetry colorimetry = {    //conversion to 1:1:7:1
    GST_VIDEO_COLOR_RANGE_0_255, 
    GST_VIDEO_COLOR_MATRIX_RGB,
    GST_VIDEO_TRANSFER_SRGB,
    GST_VIDEO_COLOR_PRIMARIES_BT709
};

[fduncanh]

@PancakeTAS 's fix is a conversion to sRGB color. THANK YOU @PancakeTAS !!

(Unfortunately needs two videoconverts in the pipeline.)

Now implemented in master branch:
b1fb510

[fduncanh]

looks like its not fixed, sorry. (was testing @PancakesTAS 's fix with GStreamer-1.25.0.1, now back to 1.24.0)

EDIT: its fixed on x86_64 (checked on Ubuntu-20.04LTS, 22.04LTS and 24.04LTS

NOT FIXED on Raspberry Pi model 4B with Ubuntu-24.04LTS and model 5 with R Pi OS bookworm.
will need more investigation to find out why.
(A slimmed-down version of @PancakeTAS 's fix was used, will next check that the full fix also is not working on R Pi)

EDIT: the full @PancakeTAS fix does not work on Raspberry Pi either.....

The fix is now an option -srgb, enabled by default in Linux, and can be disabled with "-srgb no". (this is because it adds extra processing in the pipeline, does not yet work on R Pi, and may be obsoleted by improvements in GStreamer-1.26.)