Streams backpressure can optimize throughput while limiting processing and memory consumption by pausing data production as early as possible in a data pipeline. @@ -130,6 +130,7 @@ The readEncodedData algorithm is given a |rtcObject| as p 1. Let |frame| be the newly produced frame. 1. Set |frame|.`[[owner]]` to |rtcObject|. 1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`. +1. If |rtcObject|'s |transform|'s |options| contains {{RTCRtpScriptTransformParameters/acceptOnlyInputCodecs}} with the value True, and the {{RTCEncodedVideoFrameMetadata/mimeType}} of |frame| does not match any codec in |transform|'s |options|' |inputCodecs|, execute the [$writeEncodedData$] algorithm given |rtcObject| and |frame|. 1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`. The writeEncodedData algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps: @@ -146,7 +147,7 @@ The writeEncodedData algorithm is given a |rtcObject| as On sender side, as part of [$readEncodedData$], frames produced by |rtcObject|'s encoder MUST be enqueued in |rtcObject|.`[[readable]]` in the encoder's output order. As [$writeEncodedData$] ensures that the transform cannot reorder frames, the encoder's output order is also the order followed by packetizers to generate RTP packets and assign RTP packet sequence numbers. -The packetizer may expect the transformed data to still conform to the original format, e.g. a series of NAL units separated by Annex B start codes. +The packetizer will expect the transformed data to conform to the format indicated by its {{RTCEncodedVideoFrameMetadata/mimeType}}. On receiver side, as part of [$readEncodedData$], frames produced by |rtcObject|'s packetizer MUST be enqueued in |rtcObject|.`[[readable]]` in the same encoder's output order. To ensure the order is respected, the depacketizer will typically use RTP packet sequence numbers to reorder RTP packets as needed before enqueuing frames in |rtcObject|.`[[readable]]`. @@ -168,7 +169,7 @@ The `transform` setter steps are: 4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`. 5. Initialize |newPipeToController| to a new {{AbortController}}. 6. If [=this=].`[[pipeToController]]` is not null, run the following steps: - 1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`.signal. + 1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=]. 2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`. 7. Else, run the [$chain transform algorithm$] steps. 8. Set [=this=].`[[pipeToController]]` to |newPipeToController|. @@ -176,13 +177,13 @@ The `transform` setter steps are: 10. Run the steps in the set of [$association steps$] of |transform| with [=this=]. The chain transform algorithm steps are defined as: -1. If |newPipeToController|.signal is [=AbortSignal/aborted=], abort these steps. +1. If |newPipeToController|'s [=AbortController/signal=] is [=AbortSignal/aborted=], abort these steps. 2. [=ReadableStreamDefaultReader/Release=] |reader|. 3. [=WritableStreamDefaultWriter/Release=] |writer|. 4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`. -5. Call pipeTo with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal. -6. Call pipeTo with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal. +5. Call pipeTo with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|'s [=AbortController/signal=]. +6. Call pipeTo with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|'s [=AbortController/signal=]. This algorithm is defined so that transforms can be updated dynamically. There is no guarantee on which frame will happen the switch from the previous transform to the new transform. @@ -361,6 +362,25 @@ dictionary RTCEncodedVideoFrameMetadata { ### Members ### {#RTCEncodedVideoFrameMetadata-members}
+ An identifier for the encoded frame, monotonically increasing in decode order. Its lower + 16 bits match the frame_number of the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]], if present. + Only present for received frames if the Dependency Descriptor Header Extension is present. +
++ List of frameIds of frames this frame references. + Only present for received frames if the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]] is present. +
+
+partial dictionary RTCRtpCodec {
+ DOMString packetizationMode;
+};
+
## Operations ## {#RTCRtpScriptTransform-operations}
The new RTCRtpScriptTransform(|worker|, |options|, |transfer|) constructor steps are:
@@ -781,6 +819,7 @@ The mimeType == "video/vp8");
+sender.transform = new RTCRtpScriptTransform(worker, {
+ inputCodecs: [vp8],
+ outputCodecs: [{mimeType: “video/x-encrypted”,
+ packetizationMode: "video/sframe"}]
+});
+
+// At receiver side.
+pc.ontrack = ({receiver}) => {
+ const {codecs} = receiver.getParameters();
+ const customCodec = codecs.find(({mimeType}) => mimeType == "video/x-encrypted");
+ if (customCodec) {
+ receiver.transform = new RTCRtpScriptTransform(worker, {
+ inputCodecs: [customCodec],
+ outputCodecs: [{mimeType: "video/vp8"}]
+ });
+ }
+}
+
+// Same worker can handle both sides.
+function work() {
+ onrtctransform = async ({transformer: {readable, writable}, {inputCodecs, outputCodecs}}) => {
+ const [outputCodec] = outputCodecs;
+ await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable);
+ function transform(frame, controller) {
+ // transform chunk
+ let metadata = frame.metadata();
+ const inputCodec = inputCodecs.find((mimeType) => mimeType == metadata.mediaType);
+
+ if (inputCodec && outputCodec.mimeType == "video/x-encrypted") {
+ encryptBody(frame, inputCodec);
+ metadata.mediaType = outputCodec.mimeType;
+ frame.setMetadata(metadata);
+ } else if (inputCodec.mimeType == "video/x-encrypted") {
+ decryptBody(frame, outputCodec);
+ metadata.mediaType = outputCodec.mimeType;
+ frame.setMetadata(metadata);
+ }
+ controller.enqueue(frame);
+ }
+ }
+}
+```
+# Frequently asked questions
+
+1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that?
+
+ A: Use multiple media types. Each will be assigned a payload type. Mark each frame with the media type they need to be packetized as.
+
+1. Q: What is the relationship between this proposal and the IETF standard for SFrame?
+
+ A: This proposal is intended to make it possible to implement the IETF standard for SFrame in Javascript, with only the packetizer/depacketizer being updated to support the SFrame packetization rules. It is also intended to make it possible to perform other forms of transform, including the ones that are presently deployed in the field, while marking the SDP with a truthful statement of its content.