Custom Scalability Structures

[aboba]

What if an application requires a custom scalability structure not supported in the defined scalability modes? For example, what if an application wants a 4:1 scalability ratio rather than 2:1 or 1.5:1?

Codecs such as AV1 support custom scalability structures, so that the encoder can be configured to support scalability structures beyond those defined in scalability modes.

Do we need a way to support scalability structures beyond those defined in scalability modes?

[alvestrand]

The existing scalability structures show clearly that the design space for scalability structures is very large, and probabliy requires something like a turing-complete language to specify all possible permutations.
I would think that an application-defined scalability structure would be best done by defining an interface that installed a name and a function to be called to make the send/size decisions for each frame - and that it's something we should experiment with before trying to standardize it.

[aboba]

There are several ways to address the need for extending the resolution ratios that do not involve support for custom scalability structures:

• Add scalability modes to the table. For example, modes featuring 4:1 resolution ratios could be added. It is even possible to add a mode where the resolution ratio is not constant, such as a mode to support simulcast encoding at 1080p, 720p and 360p (e.g. "S3T1p").

• Allow layers to be disabled. For example, if a mode with 3 spatial layers were selected and the middle layer were to be disabled we would have a 4:1 ratio (for S3T1) or a 3:1 ratio (for S3T1h).

[aboba]

Related to Issue #14

[alvestrand]

In S3T1, doesn't the top layer depend on the middle layer?

[jan-ivar]

TPAC ACTION: aboba to add a paragraph to the table to describe considerations for adding new values to the table of modes.

[aboba]

With the merger of PR 27, this issue has largely been resolved.

[aboba]

Reopening issue due to a request for support of "flexible mode" in WebCodecs.

[aboba]

At TPAC 2024, a proposed encoding API was presented that would enable arbitrary scalability modes in WebCodecs.