Chunked Encoding for incremental building of large objects

[GrumpyOldTroll]

Consider for example a transport system that supports the architecture described here for parallel video encoding: http://techblog.netflix.com/2015/12/high-quality-video-encoding-at-scale.html.

Imagine that the ingress for a live video processing stream could be chunked and handled in sections, and the egress talks to the ingress to find information about where to fetch the chunks being processed before passing them along:

[ { “sr”:[ “http://domain1.example.com/foo_chunk1”, “http://domain2.example.net/bar_chunk1”, ...] }

, {“sr”:[ “http://domain3.example.com/foo_chunk2”, “http://domain4.example.net/bar_chunk2”, ...] }

]

(indicating that the client should fetch http://domain1.example.com/foo_chunk1 with http://domain2.example.net/bar_chunk1 as a fallback, and then fetch and append http://domain3.example.com/foo_chunk2 with http://domain4.example.net/bar_chunk2 as a fallback, and the result of concatenating those 2 objects would be the final object.)

(Note that there are other possible encapsulations as well, this is just the simplest example that occurred to me that satisfies the request I was making.)

For the case of live video processing, chunking support with good latency would be so important that we would have to work out a system to ensure we had that, so that oob without support for chunking would not be helpful here (or would at least require a substantial part of the transport system to have specialized processing).

By contrast, the oob draft with support for chunking in a style like this would provide a significant new capability to control the distribution, failover, and redundancy for transferring this kind of live video stream through processing to an egress, with some really useful latency characteristics. Oob with this extension would save us a lot of application chatter and engineering time in negotiating these various pieces, and implementing them separately. (Or alternately, I guess an external management system to recombine chunks instead of maintaining them as part of the same stream.)

This also might have very useful applications in ad-insertion, where sections of video often would come from a different category of servers than the main video stream, especially for something like mp4 streaming that’s not already segmented.