how about support other flux checkpoint

[xueqing0622]

only Original Flux model, how about support other flux checkpoint

[lodestone-rock]

PRs are welcome
im working on something else atm

[blepping]

@xueqing0622 #14 should enable support for other Flux checkpoints. Feel free to try it, it could benefit from more testing!

[xueqing0622]

@xueqing0622 #14 should enable support for other Flux checkpoints. Feel free to try it, it could benefit from more testing!

thx, blepping, can you give a workflow for your brach
https://github.com/blepping/ComfyUI_FluxMod/tree/feat_various_improvements

[xueqing0622]

seems can't use lora

[xueqing0622]

how about use gguf unet

[blepping]

thx, blepping, can you give a workflow for your brach

There's no difference in the workflow unless you want to use advanced samplers (like SamplerCustom) and would have needed KSamplerMod. If so you can just do something like this:

And then just connect that to SamplerCustom or whatever you are using.

seems can't use lora

You'll get a bunch of warnings in the console but they seem like they at least do something. How well it works will depend though. The modded Flux is missing a bunch of layers that the LoRA may be trying to override, or maybe it will just be applying to the wrong ones. Not sure.

I tried an illustration style LoRA and it did seem to influence the generation to look more like an illustration.

how about use gguf unet

These changes may make implementing that easier but it's probably still fairly difficult to do. It's something I may mess with later, if that pull actually gets merged.

[xueqing0622]

blepping, thanks for the answer,
with lora I did get a bunch of warnings in the console that didn't work.
But it's good enough to be able to use other checkpoints.I don't know if merging lora into checkpoint will work.
Thanks for all the hard work

[blepping]

with lora I did get a bunch of warnings in the console that didn't work.

You will get warnings for the layers that don't exist in the modded version of Flux but the LoRA may still apply to the others. In other words, it may still work to some degree even if parts of the LoRA didn't apply.

You can try to use a LoRA and see if you like the results. Pretty much all I can say is it does something. Just for example, no LoRA:

With an illustration style LoRA:

With a painting style LoRA:

But it's good enough to be able to use other checkpoints.I don't know if merging lora into checkpoint will work.

It's worth a try. From what I understand, a chunk of the model in the middle is missing.

Imagine something like this with normal Flux.

0 1 2 3 4 5 6 7 8 9
  ^ ^     ^
  | |     |
  \-------- LoRA

But then we delete blocks 2, 3, 4:

0 1 5 6 7 8 9
  ^ ^     ^
  | |     |
  \-------- LoRA

The LoRA still hits 1 as intended, but the rest of the model is shifted so the part that was supposed to hit 2 now hits 5. This is just my guess, not sure that this explanation is correct.

If this is roughly correct then really the best we could do is skip parts of the LoRA that want to apply to layers that no longer exist and offset the others so they are at least hitting the intended layer. This would be fairly difficult to do and it's also pretty likely that the LoRA still wouldn't work properly (since it was trained on a different model and we might have to just skip parts of it entirely).

[xueqing0622]

blepping, I added it to the lora loader and it didn't work,
but merge it into the checkpoint was fine.
Thanks for modifying the branch, there is no checkpoints limitation, so I can merge the lora into the checkpoints to make sure it works!
Of course, it would be better if it would work in the lora loader, but unfortunately a lot of lora doesn't work for me here.

[blepping]

不客气啊！把LoRA融合进模型里面我觉得是现在的最佳的办法，希望只是一种暂时的限制但此时我不能承诺什么。有空的话我再想想怎么解决这个问题。

Seems like the pull got merged so you can switch back to the main branch. It's encouraging that merging the LoRAs seemed to work for you, trying to fix it so they at least apply to the correct (existing) layers should probably lead to similar results.

[xueqing0622]

我不会编程，只是个建议：能否把改变模型类型放在载入checkpoint和lora之后，让checkpoint包括lora载入完成之后，才去改变模型类型，

I don't know how to write script, just a suggestion: could you put changing the model type after loading the checkpoint and lora, so that the checkpoint including the lora is loaded and then you go to change the model type and quant mode to flux mode.

[blepping]

That approach would definitely be best from a user perspective, unfortunately I think it would be hard to implement. Probably the simplest way to to explain it would be to say that after you load the model with Load Diffusion Model it's been built from the wrong components: we need to start with the correct ones instead of trying to fix it afterwards.

I have an idea for a different approach which I'll try when I get a chance, it should let you apply LoRAs normally and they should work about the same as when you tried to merge them into the checkpoint.

[xueqing0622]

That approach would definitely be best from a user perspective, unfortunately I think it would be hard to implement. Probably the simplest way to to explain it would be to say that after you load the model with Load Diffusion Model it's been built from the wrong components: we need to start with the correct ones instead of trying to fix it afterwards.

I have an idea for a different approach which I'll try when I get a chance, it should let you apply LoRAs normally and they should work about the same as when you tried to merge them into the checkpoint.

Of course, I'm a non-professional.
It would be great to make sure lora works .
merge lora to checkpoint that's too time consuming and inflexible.
Looking forward to your works.
Thx , have a nice day.

[blepping]

Actually, I think my explanation about the missing blocks was wrong (at least for normal FluxMod, might be sort of correct when using the mini version). From looking closer at how this stuff works, it doesn't seem obvious why merging a LoRA into a checkpoint would work better than just trying to load it normally. It seems like the warnings when trying to load the LoRA only involve parts of the model that get deleted in FluxMod so that should be the same either way.

[xueqing0622]

use gguf still not work by load lora, only merge lora to checkpoint work.
btw, what is this lite patch ckpt name mean, do you have link for this lite ckpt

[blepping]

use gguf still not work by load lora, only merge lora to checkpoint work.

#15 only adds GGUF support, it doesn't do anything to improve the LoRA situation. LoRAs will also probably be even less likely to work if you use the lite patch.

[xueqing0622]

thx your answer, gguf and other checkponit support is very useful for my workflow.
good job!