SD3.5 - Workaround for Do not promote FP8 error

[saunderez]

Workaround the FP8 Do Not Promote error by casting to FP32 first.
Check both tensors are on the same device before using.

[saunderez]

It's slow but it works, as you can see I'm using the FP8 version of SD3.5 Large.

Result

[VeteranXT]

There is SD3.5 medium version of it. I do recommend it to use it.

[andy8992]

What's left to add 3.5 medium support... this is needed

[VeteranXT]

Hype Train!

[andy8992]

I check this dang github page every single day hoping there'll be news about this

[saunderez]

When I posted this workaround I figured someone would fix it properly in matter of days. I was mucking around with SageAttention which magically fixed the slow generation, and I would've added that here too but its too hacky. Attention needs to be split out so it's all handled in the one place, back in the Auto1111 Dreambooth days I made a wrapper function for the model that worked well but not sure if that's feasible with this implementation.

[likelovewant]

by changing float32 to float16 , the speed will catch up and works.

    x_embed = self.x_embedder(x).to(torch.float16)
    pos_embed = self.cropped_pos_embed(hw).to(torch.float16).to("cuda")
    x = x_embed + pos_embed

[saunderez]

by changing float32 to float16 , the speed will catch up and works.

    x_embed = self.x_embedder(x).to(torch.float16)
    pos_embed = self.cropped_pos_embed(hw).to(torch.float16).to("cuda")
    x = x_embed + pos_embed

Yeah to be honest I just made it cast to FP32 because it was most likely to just work. I just wanted to try out the model while I waited for official support so when the first thing I tried worked it was enough to satisfy my curiosity.

[VeteranXT]

How did you made it work?

[likelovewant]

by changing float32 to float16 , the speed will catch up and works.

    x_embed = self.x_embedder(x).to(torch.float16)
    pos_embed = self.cropped_pos_embed(hw).to(torch.float16).to("cuda")
    x = x_embed + pos_embed

Yeah to be honest I just made it cast to FP32 because it was most likely to just work. I just wanted to try out the model while I waited for official support so when the first thing I tried worked it was enough to satisfy my curiosity.

Understood,It’s so great you can find a quick workaround to get FP8 support, even if it's a temporary turnaround fix. Thanks for sharing your solution. Now, let's keep our fingers crossed for the Forge official SD3.5 medium release. I've run a few tests, but unfortunately, without much luck in getting it to work. Let‘s hope Lllyasviel will be able to make it happen soon, despite his busy schedule." @saunderez

[likelovewant]

How did you made it work?

simply mannuly change the code https://github.com/lllyasviel/stable-diffusion-webui-forge/pull/2220/files
@VeteranXT

[VeteranXT]

i did that i get "cant recognize model..."

[likelovewant]

i did that i get "cant recognize model..."

First , git checkout sd35 , edit the files , except changing float32 to float16 , downlaod necessary file for SD35 ,eg ,extra clip-g,t5xxl_fp8_e4m3fn.safetensors``sd3.5 large-fp8 models . then start the program. as showed #2183,
#2161 (comment)

@VeteranXT

[danilomaiaweb]

Here not work for me. See my forge version:

app: stable-diffusion-webui-forge.git
updated: 2024-12-10
hash: e073e4e
url: https://github.com/lllyasviel/stable-diffusion-webui-forge.git/tree/main

My branch is origin/main
How make full integration of SD3.5 on my version?

I have than chance my branch by sd35 (main...sd35) ?
Thank in advance

[likelovewant]

Here not work for me. See my forge version:

app: stable-diffusion-webui-forge.git updated: 2024-12-10 hash: e073e4e url: https://github.com/lllyasviel/stable-diffusion-webui-forge.git/tree/main

My branch is origin/main How make full integration of SD3.5 on my version?

I have than chance my branch by sd35 (main...sd35) ? Thank in advance

simply
git checkout sd35
and do the edit . launch it . if there is any error and cannot figur it out ,it's better to has error log paste here . otherwise ,nobody knows what's going on .
@danilomaiaweb

[VeteranXT]

Well im using AMD forge . Fork of this. I did edit files but all i get is --can't recognize model.

[likelovewant]

Well im using AMD forge . Fork of this. I did edit files but all i get is --can't recognize model.

test worked on AMD GPU in Zluda way ,it works. perhaps need update your huggingface guess #2161 (comment)
@VeteranXT