Use refiner with SD1.5 models

[df2df]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

Set VAE to Auto and sometimes it would work correctly but on a 2nd try after it would generate fried images similar to incorrect VAE selection which I think is what's happening. Was testing it while using Ultimate upscale and sometimes 1 tile would be fried while others were good. Subsequent tries led to all tiles being scrambled and continues to get worse.

May be related to this issue which is something to do with XL models somehow polluting the memory even after unloading and loading new checkpoints.

Settings: no caching of VAE, 2 checkpoints cached with 1 one on device selected

Steps to reproduce the problem

1. Select SD1.5 model, enable refiner in tab and select XL base refiner
2. Set to Auto VAE option
3. Generate normally or with Ultimate upscale

What should have happened?

Uncorrupted images

Sysinfo

Linux Ubuntu 22.04 6800XT 32GB ram

What browsers do you use to access the UI ?

No response

Console logs

No errors

Additional information

No response

[w-e-w]

you're not supposed to be able to mix SD1.5 with XL refine
the latent space of SD1.5 and SDXL are not the same

[w-e-w]

someone seems to be interested in making this possible
but it's not working natively it's not bug
so I reopened and labeled this as enhancement

[BlandiZ]

in original Fooocus latest release from Ilia, Fooocus_win64_2-1-754 based on CUDA 12 and Pytorch2.1, there is a possibility to use SDXL as base and SD1.5 as refiner so the opposite should be possible as well (technically)? not in fooocus thou but in 1111 or ComfyUI?

[logicmoon]

In ComfyUI this is also possible through workflows or extensions (see links). It would be great to have this feature in A1111. Until supported, we will have to manually use img2img I guess, but thank you for looking into this.

[w-e-w]

I knew about SD-Latent-Interposer
w-e-w/sd-webui-hires-fix-tweaks#3
but implementing it in webview is not a straightforward

[SmirkingKitsune]

I think what you are describing is what the sd-webui-refiner extension does.
It seems to have a large VRAM requirement, so it is not suitable for low VRAM systems (<12 GB VRAM), but since you have an RX 6800XT (16 GB VRAM), it should work for you.