Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a section for the background blur field in VideoFrameMetadata #159

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add a section for the background blur field in VideoFrameMetadata #159

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a1077cb
Add a section for the background blur field in VideoFrameMetadata
Nov 12, 2024
2e42d6a
Address PR comments
Nov 15, 2024
5c7c78f
Address PR comments
Jan 14, 2025
413ab3d
Update index.html
brychanrobot Feb 10, 2025
3cffcc5
Adjust blur effect status description
brychanrobot Feb 10, 2025
cef4b7a
Be more specific about what it means if blur field isn't present
brychanrobot Feb 10, 2025
cafc996
Adjust wording of backgroudn blur a bit more to explain missing property
Feb 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1876,6 +1876,85 @@ <h3>Examples</h3>
</pre>
</section>
</section>
<section>
<h2>Background Blur Effect Status</h2>
<p>The {{VideoFrameMetadata}} interface exposes the blur state as a property,
which allows apps to know the state for every frame. This is important for
scenarios where the app must ensure user privacy by never sending an
brychanrobot marked this conversation as resolved.
Show resolved Hide resolved
un-blurred frame off of the user's device.</p>
brychanrobot marked this conversation as resolved.
Show resolved Hide resolved
<section>
<h3>{{VideoFrameMetadata}}</h3>
<pre class="idl">
partial dictionary VideoFrameMetadata {
BackgroundBlur backgroundBlur;
};</pre>
<section class="notoc">
<h4>Members</h4>
<dl class="dictionary-members" data-link-for="VideoFrameMetadata" data-dfn-for="VideoFrameMetadata">
<dt><dfn><code>backgroundBlur</code></dfn> of type <code>{{BackgroundBlur}}</code></dt>
<dd>
<p>The state of the background blur effect for the current frame. If
the property isn't present, it indicates that the user agent doesn't
brychanrobot marked this conversation as resolved.
Show resolved Hide resolved
support reporting blur state.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it supposed to be present on VideoFrames from non-camera sources? The text seems to assume every VideoFrame has a background blur effect state, when AFAIK only cameras have this? Are we to intuit {backgroundBlur: false} in those cases? If so, we might want to clarify that.

What's the precedent here for metadata specific to only certain sources?

I've opened w3c/webcodecs#879, but unless there's a precedent (which I couldn't find), it seems better to me for it to only be present in VideoFrames from cameras. That way false represents a user setting,.

This way, downstream code can differentiate false that from absence (unknown) and make a more informed decision about transmission.

This would have the effect of revealing that the VideoFrame came from a camera, but I can't think of a situation where that would be problematic.

How about:

Suggested change
<p>The state of the background blur effect for the current frame. If
the property isn't present, it indicates that the user agent doesn't
support reporting blur state.
<p>The state of the background blur effect for the current frame, if
[=map/exist|present=]. Absence might indicate that the frame is not
from a camera, or the user agent might not support reporting blur state.

?

I've softened it a bit with "might" since the VideoFrame has a constructor, which means presence and values might have more to do with what metadata was passed in by JS.

</p>
</dd>
</dl>
</section>
</section>
<section>
<h3>{{BackgroundBlur}}</h3>
<pre class="idl">
dictionary BackgroundBlur: MediaEffectInfo {};</pre>
<section class="notoc">
</section>
</section>
<section>
<h3>{{MediaEffectInfo}}</h3>
<pre class="idl">
dictionary MediaEffectInfo {
required boolean enabled;
};</pre>
<section class="notoc">
<h4>Dictionary {{MediaEffectInfo}} Members</h4>
<dl class="dictionary-members" data-dfn-for="MediaEffectInfo" data-link-for="MediaEffectInfo">
<dt><dfn><code>enabled</code></dfn> of type {{boolean}}</dt>
<dd>
<p>True if the effect is enabled, false otherwise. This isn't a strong guarantee, as user
agents likely can't detect all possible video manipulation software.
</p>
</dd>
</dl>
</section>
</section>
<section>
<h3>Example</h3>
<pre class="example">
const stream = await navigator.mediaDevices.getUserMedia({ video: true });
const videoTrack = stream.getVideoTracks()[0];
const blurIndicator = document.getElementById("blurIndicator");
const trackProcessor = new MediaStreamTrackProcessor({ track: videoTrack });
const trackGenerator = new VideoTrackGenerator({ kind: "video" });

const transformer = new TransformStream({
async transform(videoFrame, controller) {
if ("backgroundBlur" in videoFrame.metadata()) {
console.log('backgroundBlur.enabled:', videoFrame.metadata().backgroundBlur.enabled);
} else {
console.log('backgroundBlur unsupported');
}

controller.enqueue(videoFrame);
},
});

trackProcessor.readable
.pipeThrough(transformer)
.pipeTo(trackGenerator.writable);

const processedStream = new MediaStream();
</pre>
</section>
</section>
<section>
<h2>MediaStream in workers</h2>
<div>
Expand Down