Skip to content

Commit

Permalink
Add WEBGL_element_texture (KhronosGroup#2508)
Browse files Browse the repository at this point in the history 
     * Add new HTMLElementTexture object and new tokens for
       TEXTURE_ELEMENT_WEBGL texture target.
  • Loading branch information
@tharkum
tharkum committed Sep 19, 2017
1 parent d48fd7f commit 6215e95
Showing 1 changed file with 298 additions and 0 deletions.
298 changes: 298 additions & 0 deletions extensions/proposals/WEBGL_element_texture/extension.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,298 @@
<?xml version="1.0" encoding="UTF-8"?>
<proposal href="proposals/WEBGL_element_texture/">
<name>WEBGL_element_texture</name>

<contact> <a href="https://www.khronos.org/webgl/public-mailing-list/">WebGL
working group</a> (public_webgl 'at' khronos.org) </contact>

<contributors>
<contributor>Byungseon Shin, LG Electronics</contributor>

<contributor>Andrei Volykhin, LG Electronics</contributor>

<contributor>Mark Callow, HI Corporation</contributor>

<contributor>Members of the WebGL working group</contributor>
</contributors>

<number>NN</number>

<depends>
<api version="1.0"/>
</depends>

<overview>
<p>This extension defines a new object, the <code>HTMLElementTexture</code>,
that can be used to efficiently transfer a sequence of image frames from
a producer which out of control the WebGL into a WebGL texture.
This is done via a new texture target, <code>TEXTURE_ELEMENT_WEBGL</code>
which can only be specified as being the consumer of an image stream from a
producer element.</p>

<p>There is no support for most of the functions that manipulate other
texture targets (e.g. you cannot use <code>*[Tt]ex*Image*()</code>
functions with <code>TEXTURE_ELEMENT_WEBGL</code>). Also,
<code>TEXTURE_ELEMENT_WEBGL</code> targets never have more than a single
level of detail. Because of these restrictions, it is possible to
allow sources which have internal formats not otherwise supported by WebGL,
such as planar or interleaved YUV data to be WebGL texture target siblings.</p>

<p>The extension extends GLSL ES with a new <code>samplerElementWebGL</code> type
and matching sampling functions that provide a place for an implementation
to inject code for sampling non-RGB data when necessary without degrading performance
for other texture targets. Sampling a <code>TEXTURE_ELEMENT_WEBGL</code> via a sampler of
type <code>samplerElementWebGL</code> always returns RGBA data.</p>

<p>Sampling a WebGL texture will return an RGBA vector in the same
colorspace as the source image. If the source image is stored in YUV
(or some other basis) then the YUV values will be transformed to RGB values.
(But these RGB values will be in the same colorspace as the original image.
Colorspace here includes the linear or non-linear encoding of the samples.
For example, if the original image is in the sRGB color space then the RGB value
returned by the sampler will also be sRGB, and if the original image is stored
in ITU-R Rec. 601 YV12 then the RGB value returned by the sampler will be
an RGB value in the ITU-R Rec. 601 colorspace)</p>

<p>Sampling a WebGL texture which is not associated with any
<code>HTMLElementTexture</code> object will return a sample value of (0,0,0,1).</p>

<p>Each <code>TEXTURE_ELEMENT_WEBGL</code> texture object may require up
to 3 texture image units for each texture unit to which it is bound.
The number of texture image units required by a bound texture object
can be queried using <code>getTexParameter</code> with target set
to the texture target in question, value set to
<code>REQUIRED_TEXTURE_ELEMENT_IMAGE_UNITS_WEBGL</code>, and ActiveTexture
set to the texture unit to which the texture object is bound.</p>

<p><code>HTMLElementTexture</code> provides a commands for <em>latching</em> an
image frame into the consuming texture as its contents and retrieving additional
information as a timestamp or a transformation matrix.</p>

<p>This extension does not cover the details of how a <code>HTMLELementTexture</code>
object could be created from a producer element.
Different kinds of producers works differently and described in additional externsion
specifications which will extend this one.
Example of producer element specification:
WEBGL_video_texture</p>

<features>
<glsl extname="WEBGL_element_texture">
<stage type="vertex"/>

<stage type="fragment"/>

<type name="samplerElementWEBGL"/>

<function name="texture2D" type="vec4">
<param name="sampler" type="samplerElementWEBGL"/>

<param name="coord" type="vec2"/>
</function>

<function name="texture2DProj" type="vec4">
<param name="sampler" type="samplerElementWEBGL"/>

<param name="coord" type="vec3"/>
</function>

<function name="texture2DProj" type="vec4">
<param name="sampler" type="samplerElementWEBGL"/>

<param name="coord" type="vec4"/>
</function>

</glsl>
</features>
</overview>

<idl xml:space="preserve">
[
NoInterfaceObject
] interface HTMLElementTexture : EventTarget {
void updateTexImage();

double getTimestamp();
void getTransformMatrix(Float32Array matrix);

double getLastAvailableTimestamp();

attribute EventHandler onframeavailable;
};

[
NoInterfaceObject
] interface WEBGL_element_texture {
const GLenum TEXTURE_ELEMENT_WEBGL = 0x9248;
const GLenum SAMPLER_ELEMENT_WEBGL = 0x9249;
const GLenum TEXTURE_BINDING_ELEMENT_WEBGL = 0x924A;
const GLenum REQUIRED_TEXTURE_ELEMENT_IMAGE_UNITS_WEBGL = 0x924B;
};
</idl>

<!-- new functions -->

<newfun>
<p>On <code>HTMLElementTexture</code>:</p>

<function name="updateTexImage" type="void">
Update the texture image to the most recent frame from the image stream
and release the previously-held frame.
This may cause some frames of the stream to be skipped.
Sampling the <code>WebGLTexture</code>,
that is the <code>HTMLElementTexture</code>'s <em>consumer</em>, will
return values from the latched image.
The image data is guaranteed not to change as long as the image is latched.
</function>

<function name="getTimestamp" type="double">
Retrieve the timestamp associated with the texture image set by the most
recent call to updateTexImage. This timestamp represent the time of frame
relative to start of the producer timeline, the time when
the frame was produced (the Media Stream Counter in OpenML),
not the time when frame was received by application.
This timestamp is in seconds, and is normally monotonically increasing.
The timestamp should be unaffected by time-of-day adjustments and
should be strictly monotonic but for some image producers (media)
may be reset when the position is set.
The specific meaning and zero point of the timestamp depends
on the source providing images to the <code>HTMLELementTexture</code>.
Unless otherwise specified by the image source, timestamps cannot
generally be compared across <code>HTMLElementTexture</code> instances,
or across multiple application invocations. It is mostly useful for
determining time offsets between subsequent frames.
</function>

<function name="getTransformMatrix" type="void">
<param name="matrix" type="Float32Array"/>
Retrieve the 4x4 texture coordinate transform matrix associated with
the texture image set by the most recent call to updateTexImage.
This transform matrix maps 2D homogeneous texture coordinates
of the form (s, t, 0, 1) with s and t in the inclusive range [0, 1]
to the texture coordinate that should be used to sample that location
from the texture. Sampling the texture outside of the range
of this transform is undefined.
The matrix is stored in column-major order.
</function>

<function name="getLastAvailableTimestamp" type="double">
Retrieve the timestamp associated with the most recent texture image
which could be provided by a producer.
</function>
</newfun>

<dl class="methods">
<dt><code class="attribute-name">onframeavailable</code> of type
<code>EventHandler</code></dt>

<dd>The <code>onframeavailable</code> handler is executed when a new frame
could be latched from image stream.</dd>
</dl>

<!-- new tokens -->

<newtok>
<p>On <code>WEBGL_element_texture</code>:</p>

<p>The meaning and use of these tokens is similar as described in <a
href="https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt">OES_EGL_image_external</a>.</p>

<function name="bindTexture" type="void">
<param name="target" type="GLenum"/>
<param name="texture" type="WebGLTexture?"/>
<code>TEXTURE_ELEMENT_WEBGL</code> is accepted as a target by the
<code>target</code> parameter of <code>bindTexture()</code>
</function>

<function name="getActiveUniform" type="WebGLActiveInfo?">
<param name="program" type="WebGLProgram?"/>
<param name="index" type="GLuint"/>
<code>SAMPLER_ELEMENT_WEBGL</code> can be returned in the
<code>type</code> field of the <code>WebGLActiveInfo</code> returned by
<code>getActiveUniform()</code>
</function>

<function name="getParameter" type="any">
<param name="pname" type="GLenum"/>
<code>TEXTURE_BINDING_ELEMENT_WEBGL</code> is accepted by
the <code>pname</code> parameter of <code>getParameter()</code>
</function>

<function name="getTexParameter*" type="any">
<param name="target" type="GLenum"/>
<param name="pname" type="GLenum"/>
<code>REQUIRED_TEXTURE_ELEMENT_IMAGE_UNITS_WEBGL</code> is accepted
as the <code>pname</code> parameter of <code>getTexParameter*()</code>
</function>
</newtok>

<samplecode xml:space="preserve">

<p> This a fragment shader that samples a element texture.</p>
<pre>
#extension GL_WEBGL_element_texture : require
precision mediump float;
varying vec2 v_texCoord;

uniform samplerElementWEBGL uSampler;

void main(void) {
gl_FragColor = texture2D(uSampler, v_texCoord);
}
</pre>

<p>This shows application that renders HTML*Element using proposed extension.</p>
<pre>
var customElement = document.getElementById("...");
var customTexture = gl.createTexture();
var elementTexture;

function initialize() {
var ext = gl.getExtension('WEBGL_element_texture');
if (ext === null)
return;

elementTexture = ext.create*Texture(customElement, customTexture);
element.onframeavailable = function() { ... };
}

function update() {
gl.bindTexture(ext.TEXTURE_ELEMENT_WEBGL, customTexture);
elementTexture.updateTexImage();
gl.bindTexture(ext.TEXTURE_ELEMENT_WEBGL, null);
}

function render() {
.....

gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(ext.TEXTURE_ELEMENT_WEBGL, customTexture);

.....

gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
}
</pre>

<p>Application renders each video frames into WebGL canvas based on game-loop pattern.</p>
<pre>
initialize();

while (true) {
update();
processInput();
render();
}
</pre>

</samplecode>

<tests/>

<issues/>

<history>
<revision date="2017/09/19">
<change>Initial revision.</change>
</revision>
</history>
</proposal>

0 comments on commit 6215e95

Please sign in to comment.