Layer Refresh

[aboba]

When sending a spatial scalabilityMode, the user agent can choose to drop a spatial layer at any time. However, adding a spatial layer is tricky, because a spatial extension layer frame may depend on a previous frame within that extension layer. If that previous frame was not received, either because the sender did not send it or because it was dropped by an SFU, future spatial extension layer frames will not be decodable.

As a result, if the sender or SFU re-adds a spatial layer it had previously dropped, either the receiver will notice that the extension layer frame's dependencies are not met (e.g. via an examination of the Dependency Descriptor) and will filter the frame, or it will pass the frame to the decoder which will result in a decoder error, causing the receiver to send a PLI. Receipt of a PLI will cause the sender to reset the encoding, sending a base layer keyframe and restarting the spatial scalabilityMode sequence. In the event that no base layer loss was experienced, this will cause a larger bandwidth spike than is necessary.

One way to address this issue would be to add an API surface (in WebCodecs, WebRTC or both) for the encoder to request a layer refresh.

Another approach is to enable receivers to request a layer refresh. The Layer Refresh Request (LRR) RTCP feedback message enables a receiver that encounters a missing dependency to request that a layer be refreshed (e.g. that the next frame issued depend only on lower spatial layers, not previous frames of the same spatial layer). This is more efficient than sending a PLI, since it avoids having the encoder send a base layer keyframe.

So far, the LRR RTCP feedback message has not been implemented in any browser. So far we have avoided a normative reference to the LRR specification in the document, in part because draft-ietf-avtext-lrr has been sitting in the RFC Editor Queue since 2017, unable to be published because of an unresolved reference to draft-ietf-avtext-framemarking. However, that logjam may soon be resolved since the framemarking document is now in IESG review.

[aboba]

Support for layer refresh will require changes to the underlying encoding APIs, so marking this as an enhancement. At TPAC 2024, we had a presentation relating to this.