Issues needing to be solved in WEBGL_video_texture proposal

[kenrussell]

In #2470 many issues were raised by @msc- @tharkum @zhenyao and others. Opening this issue to capture them separately from that pull request.

In discussions with people working on Chrome Media and color-correct rendering in Chrome, the point has been made that the implicit conversion to an "RGB" color space, which will occur while sampling the video, should define that the result is extended sRGB. This handles wide gamut and HDR videos. Canvases are increasingly color managed in browsers, so the conversion from sRGB to the display color profile will be done by the browser's compositor afterward.

There are many more issues to be discussed. Please add to this discussion and propose pull requests addressing portions of them.

[MarkCallow]

OES_EGL_image_external says that the RGB samples are in whatever color space the video is in giving the examples of sRGB and Rec. 601. Do you really want to convert everything to extended sRGB or is it better to have a query so the app can determine the color space?

If you're willing to do conversion, maybe it is better to return samples in linear RGB space. That would make life easier for applications.

When I have time I'll copy my list of issues from #2470 to here. As for the HTMLElementTexture proposal, I'm on the fence about adding that here also or opening another issue. Opinions?

Speaking of that, @kenrussell wrote

however, video elements are handled very differently than other DOM elements, and I think we are going to need special handling for them.

I think at the level of OES_EGL_image_external and related extensions, which is very generic, there is no difference. It gives you a texture object which is an opaque block of memory whose content is written by an external agent and could be updated at any time. To avoid tearing you need either (a) multiple buffers, e.g. an EGLStream, or (b) the application has to explicitly synchronize using locks and controlling the operation of the external agent. (a) is completely generic. Controlling the agent in (b) is specific to the agent type but is outside the scope of the external texture spec. Locking is also outside the scope.

Please give some examples of how the differences between these different DOM element types leak into the external texture seen by WebGL?

[RafaelCintron]

Edge currently uses the MediaEngine APIs for video playback. Specifically, for Canvas (2D and WebGL), it uses the TransferVideoFrame API to extract video frames.

TransferVideoFrame does more than just "copy" the video into the output texture. Depending on the video, it also performs one or more of the following operations:

• On-the-fly rotation for flipped videos
• Stereo video processing (extracting a single view)
• Deinterlacing of interlaced videos
• Handling of square pixels
• Color space conversions - combining data from two textures into one.

Is the plan to automatically do these conversions when the web developer "samples" from a video texture? If so, I don't think we can avoid on-the-fly shader compilation in ANGLE to make this work.

Another potential problem is developers that want to use a custom filtering shader that contains multiple video texture samples per pixel. Depending on the video (and how many of these automatic operations can occur) the developer might be better off doing a draw into an intermediate and then doing the custom filtering on the intermediate. The extension, as it is written, doesn't provide the developer with an indication of the potential penalty.

[MarkCallow]

OES_EGL_image_external says that the RGB samples are in whatever color space the video is in

I'm pretty sure OES_EGL_image_external was written before OpenGL ES had sRGB rendering and when color space management in it was like the Wild West. For correct blending the RGB samples must be in linear space so I suggest whatever extension we end up with makes it so.

[MarkCallow]

@RafaelCintron, the kind of processing you describe was envisaged when OES_EGL_image_external was written which is why the format was made opaque, samples are returned in RGB and use of multiple texture units may be required.

I agree that shader recompilation may be unavoidable, depending on the underlying hardware, and that it is important for the extension to indicate the potential penalty and when it will happen. This is one reason I so dislike the current proposal's overloading of VideoElementTargetVideoTexture() for both setting up the texture object representing the video and latching the latest frame. The former operation could trigger shader recompilation, if the shaders have already been compiled.

As an example, if an app, thinking this function is lightweight, were to naively alternate between 2 HTMLVideoElements with different formats on a single texture object it could end up recompiling the shaders every frame!!

[tharkum]

@msc- @kenrussell @xingri
I tried to combine all our discussions into one piece - please refer to #2511

I will try to rewrite WEBGL_video_texture on top of based WEBGL_element_texture extension with partial interface option soon.

May be not moments were properly described, but lets use it as start point.
I welcome your feedback.

[kenrussell]

@FredrikHubinette , Chris Cameron and I talked today about color spaces as they would relate to the WEBGL_video_texture proposal. Notes are here:
https://docs.google.com/document/d/e/2PACX-1vSH67bm1c1yTN04w5AJ0mRok7yDMFKpAdsWwXEU1BphA73b03i2zgf-DHwpHiwq31921hK84tC1LfG7/pub

Please request access if you would like to comment on the doc.

Fredrik and Chris have done a lot of work on Chrome's media stack and color correct rendering paths, and had excellent advice on the per-frame metadata that should be returned from the WEBGL_video_texture extension, as well as the color space in which the video textures should be sampled.

I'm strongly opposed to creating a second WEBGL_element_texture extension per #2511. There are many video-specific concerns to be addressed, and I think it's premature (and very likely ill-advised even in the long run) to try to handle both videos and "arbitrary DOM elements" the same way. Let's get the video case right first. It's non-trivial, but at least it's well scoped, and browsers are already handling them well natively. If we can get the video-to-WebGL-texture path working as well as the browser's native compositor does, we'll already have improved things a lot.

[MarkCallow]

A couple of things re. the color space discussion,

• canvas color extensions that junov@ and reza are working on

  they allow sRGB, scRGB-linear, extended-sRGB, P3, BT2020
  the shader output is the canvas color space.

I presume this means the color values written by the shader should be in the canvas color space rather than expecting the hardware to apply the transfer function. Is anyone working on extensions for any of the graphics APIs to do the equivalent of OpenGL's sRGB rendering for, e.g. extended sRGB so the shader could just output linear values?

so: chris thinks having the sampled input be in the output color space is the correct thing.

• redrik:

things like blending won't work correctly.
chris: no solution for this...

There are a couple of non-universal solutions. If a framebuffer fetch extension is available, the shader can do the blending with appropriate conversion to linear and back, provided there's a query so it can determine the transfer function to use. Or sRGB-texture- and sRGB-rendering-like extensions can be developed for extended sRGB. In both these cases it would better to have the samples in linear space.

[MarkCallow]

My searches are failing to turn up any specification for "extended-sRGB". Is it actually scRGB-non-linear? Some Android documentation suggests that.

[FredrikHubinette]

Yeah, it's sRGB, where the curve continues to infinity, and the negative side mirrors the positive side. I'm not sure what the *correct* name for this is, but "extended sRGB" seems to be the most descriptive.

…

On Wed, Sep 20, 2017 at 3:54 PM, Mark Callow ***@***.***> wrote: My searches are failing to turn up any specification for "extended-sRGB". Is it actually scRGB-non-linear? Some Android documentation suggests that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYT-iLISQu5h0gQqc1fYSXN5I_KUWTUgks5skZeegaJpZM4PZ-V-> .

[MarkCallow]

For the WG's consideration in Chicago..

I feel strongly that the samples returned from a video texture should be linear. Otherwise they are different from all the other color samples an application's shaders will be dealing with. If the app wants to light the video, use the sRGB rendering hardware (i.e write linear values out) or a myriad of other things, it will need linear. I thing the majority of apps will want linear so it is better if the extension provides it.

I also think the notion that the extension can provide the samples in the output color space is impractical. The WebGL implementation would require an immense amount of tracking and, e.g, what happens when the app switches FBO?

If there truly is a reasonable number of apps that would like to get the non-linear values, then a parameter could be added to the video texture creation function or an attribute to the video texture object to allow the app to choose.

When reading the minutes of the Google team discussion I sometimes found myself wondering if the participants remembered that the samples in the shader are in floating point format.

[MarkCallow]

One more thought for @kenrussell on this

I'm strongly opposed to creating a second WEBGL_element_texture extension per #2511. There are many video-specific concerns to be addressed

The iframe texture proof of concept demo that @zhenyao showed in the Siggraph BoF used a page from YouTube with a playing video. So are iframe and video elements really that different? It seems very likely that similar functionality will be needed in both case to keep audio in sync.

[kenrussell]

I feel strongly that the samples returned from a video texture should be linear. Otherwise they are
different from all the other color samples an application's shaders will be dealing with. If the app
wants to light the video, use the sRGB rendering hardware (i.e write linear values out) or a myriad
of other things, it will need linear. I thing the majority of apps will want linear so it is better if the
extension provides it.

The intent is certainly to match the behavior of the application's other rendering. To be honest I don't know the current state of Chrome's color correct rendering work, so have to defer to @FredrikHubinette and others. If the WebGL-rendered canvas is considered to be writing linear values to its color buffer and then, upon output to the display, those are conceptually converted to sRGB – which I think is what Safari does – then I agree the values sampled from the video should be in a linear color space.

When reading the minutes of the Google team discussion I sometimes found myself wondering if
the participants remembered that the samples in the shader are in floating point format.

We remembered this. The notes were quite raw and I didn't have time to clean them up.

The iframe texture proof of concept demo that @zhenyao showed in the Siggraph BoF used a
page from YouTube with a playing video. So are iframe and video elements really that different?
It seems very likely that similar functionality will be needed in both case to keep audio in sync.

The two use cases are different. Videos consistently produce frames, while arbitrary web page rendering does not. Arbitrary web page rendering is considerably more expensive, making a synchronous API impractical. I think we should focus on getting the video use case right at first.

[greggman]

I'm sure this should probably be a separate issue but just incase someone has an idea that fits the current proposal, it sure would be nice if video could be used in a worker with offscreencanvas somehow.

Like one idea is some proposal that you could call videoSource = videoElement.getContext("video-source") or some kind of object that you could transfer over to a worker.