🍰🎞️ Tiny AutoEncoder for Stable Diffusion Videos

What is TAESDV?

TAESDV is a Tiny AutoEncoder for Stable Diffusion Videos. TAESDV can decode sequences of Stable Diffusion latents into continuous videos with much smoother results than single-frame TAESD (but within the same tiny runtime budget).

Since TAESDV efficiently supports both parallel and sequential frame decoding, TAESDV should be useful for:

1. Fast batched previewing for video-generation systems like SVD or AnimateLCM.
2. Fast realtime decoding for interactive v2v systems like StreamDiffusion.

Original Video	TAESD Encode, TAESD Decode	TAESD Encode, TAESDV Decode

Note

Lots of TODOs still:

1. Add StreamDiffusion or other v2v example
2. Add performance metrics (it's like the same as TAESD)
3. Better / more example videos
4. Add to Diffusers somehow?
5. Even better checkpoint?

How can I use TAESDV for previewing generated videos?

See the AnimateLCM previewing example, which visualizes a TAESDV preview after each generation step.

demo_notebook_vis_smol.mov.compressed.mp4