Latent noise manipulation with perlin noise + upscale

[ntdviet]

This is from the discussions here: #853 (reply in thread)
and here: city96/SD-Advanced-Noise#1
...if @Extraltodeus and @city96 want to continue

Guys, I hope you have something. I give up. Couldn't make it work for the SDXL Base+Refiner flow.
The latent upscaler is okayish for XL, but in conjunction with perlin noise injection, the artifacts coming from upscaling gets reinforced so much that the 2nd sampler needs a lot of denoising for a clean image, about 50% - 60%. And that throws the image's composition around. I tried without leftover noise, with leftover noise, noise injection before upscaling, after upscaling. Can't find the sweet spot for noise injection strength and denoise schedule to work together.

Samples--
Too much denoised:

Kind of right denoise amount, but too much noise injection, so the white&black spots appear:

1 step-back for Sigma calculation and it is already not enough noise injected

And how the hell can this happen? Not enough noise for everything else but the face is ultra detailed

[Extraltodeus]

I will give you a more detailed answer later but for now my best guess are:

• maybe doing 10 rewinded steps with the base model before the refiner since two different models don't seem to work.

• you can slerp the latents in between not enough and too much to adjust more finely.

• lower the perlin persistence to get less dots.

[ntdviet]

Thanks for reply.
Your bullet no1 and no2 doesn't work out. I tried already too many combinations over the last 2 nights. Overlapped sampling, sampling gaps (rewind, ff in your language), over-, under-strength for noise... I feel like you don't have the luxury of finetuning the slerp since it's 2 different models AND the latent upscale is messing with injected noise. Basically, the more unfinished the image is from the first sampler, the more the 2nd sampler needs to denoise, hence the big composition change. The more finished the image is from first sampler, the less the second sampler needs to denoise, reducing the risk for mutations, but also reducing the amount of noise we can inject to make an improvement. It is so easy to do this from encoded pixels, but the latent upscaling, especially with XL latents, make this so damn difficult.
I believed there must be a golden set of values to achieve the right balance, but I couldn't find it, and I gave up :))

No3 - Ok, good to know, thanks. Will try with lower persistence, cause I already got to a point where I think the resulting image is not too bad, except that the fine details got overblown and the white and black spots are increasing quickly with every tiny noise strength increase.
See here:

[Extraltodeus]

I have not yet found the right settings but I'm getting closer. The idea is to rewind a few steps back with the base model itself after the upscale since using two different models for doing just never works. As well as in general, rewinding the refiner over the base model never ever gives anything good. At least to my experience.

The perlin based noise is still a little too strong and with SDXL gives me variables results depending on the subject. Kentucky houses are always crumbling down. Probably due to the perlin noise since swapping to usual noise after the upscale makes things smoother and definitely not as broken.

I intend to rework the way I generate this noise.

My Comfy carbonara:

You will need this small set of nodes that I used as sliders.. The last image in these examples contains the most advanced version of the workflow.

With:

Without:

A nicer attempt with lower persistence:

Some bottled galaxies.

With:

Without:

With:

Without:

Depending on the context the extra intensity coming from the noise can be a nice source of details. Yet also destructive for photorealistic images.

Edit: omg indeed (40steps in total):

[Extraltodeus]

I can't copy the link because I am on my phone but it is in my previous answer. It's a bunch of dynamically generated nodes for debugging.

[city96]

Thanks! Thought I had installed that, guess I deleted it when I was debugging my own nodes lol.

[williamcboehmjr]

I'm confused by some of this. I get the noise injection but this part.. are you training a vae? I'm from A1111 but have been using comfy ever since SDXL came out and I love it. I love it just because I now understand the technology a bit better than before. When A1111 you would just grab extension to do the thing a bit better... never fully understanding what's happening under the hood. With Comfyui you build the engine or grab a prebuilt engine and tinker with it to your liking. I'm loving it. Maybe a bit obsessive and lately I've been tryin not to spend hours upon hours playing with my spaghetti, lol. But anyway, is this a trained vae? Lol

[city96]

I'm confused by some of this. I get the noise injection but this part.. are you training a vae?

Not quite, though I did try to do that too lol (Don't use it, it was mostly just a proof of concept).

Anyway, the current thing I'm working on is a tiny model that tries to upscale the latent properly. The repo is here if you want to check it out. I'm mostly using it between two samplers for a "hires fix" at a very low denoise. Extraltodeus asked me to make it originally for the noise injection stuff, hence why I'm posting it here as well.

Here's an example of what happens when you upscale a latent normally with the default node. This is the reason why you usually need denoise 0.4+ when doing a second pass (or "hires fix").

With Comfyui you build the engine or grab a prebuilt engine and tinker with it to your liking. I'm loving it.

I'm mostly loving it for the rapid prototyping aspect. You can throw together a custom node a lot easier than an A1111 script, at least in my opinion.

[Extraltodeus]

@city96 I just tested your latest latent upscaler for SDXL and the quality is so much better than the first! Congrats!!

[Extraltodeus]

Best result so far is by rewinding 10 steps with the base model before using the refiner (35 steps in total):

[ntdviet]

@Extraltodeus : It's not working with realistic human photos (my focus of work lately). Even though you already circumvent the issue of interference (from latent upscale and noise injection) amplifying the noise by separating the final sampler from the upscale step.
See here, I used your wflow:

Anyways, I need to solve the problem with 2 samplers, preserving the 3rd or 4th pass for other means and not just refining. So back to the 2 sampler wflow. What I found out is I can reduce the noise (by lowering persistence, sigma strength, or increasing the sampling steps) from base without loosing much detail level in the end image. So I scrapped perlin noise injection for base sampler completely and focus on tweaking the 2nd pass, hence having a much better control over the composition (0.4 denoise for refiner like you do can still mess up image composition. 0.4 is not a problem if 2nd sampler use the same model, but different models already have big changes since 0.35)

I came to this result which is the best so far.

I found that by increasing noise iteration to 8, the resulting noise got smoothed out and I can crank up the persistence to 1.45 and get much finer details without the risk of black and white dots. (If you wonder why I calculate Sigma from step 51 to 60 and not in sync with sampler, step 45 to 60, it's because it would introduce new details with feathers growing wildly all over the body

So now, my only question is, how do I tweak the Perlin Noise injection, so that the patterns are best preserved. You see here the original base sampler give me a jewellery top, which gets warped into a feather costume in the image above. Probably because the noise fractals resemble feathers in the eye of the refiner.
Orignal base sampler pix:

[Extraltodeus]

Check out the latest version!

[williamcboehmjr]

Loving the work, you guys need to start a workflow release on civitai! That way when things get updated so does the workflow. Right now all the workflow posted in the comments thread don't work with the updated nodes it makes some of the controls say nAn

[Extraltodeus]

@ntdviet @city96 I rewrote the perlin generator and it is now 100% stable and a lot easier to use!

I little git pull or re-download of the node should get you rollin' :D

[city96]

Good job, that's a lot more user friendly!

Looks like the reproducibility issues are gone too. I slapped it into a random workflow to quickly check and it seems to keep coherency/add detail as expected. (Sorry for the crap quality screencap lol).

Probably not the proper usecase but I had it loaded so I thought why not. And hey, at least it acts as a torture test to see if it works with random weird resolutions. Can confirm that that problem is solved as well.

[Extraltodeus]

@city96 Thanks! By the way.... I rewrote a tiny bit of it again!

Now for sure the patterns matches (I had quite some doubts with the ~1h ago version)

If you know the crosseye method you can see that the patterns matches now.

It is also more efficient because I only generate one layer per channel instead of X which was quite a CPU smoker.

I can confidently say that it is now perfectly fitting what I intended while staying simple.

Using a lanczos downscale makes it more obvious:

[ntdviet]

I think I got it this time. Thanks to @Extraltodeus and @city96 for your great updates.
Found out perlin noise is more creative (means introduce new details where there was none) and standard noise is more detailing while keeping the old pattern. So I mixed both for 2nd pass. Also scrapped the SDXL refiner since base model + noise + cfg upscale already does the job without giving me deformed body parts (at 50% denoise) . I feel like a noise-dj atm :)) It was fun!

[Extraltodeus]

standard noise is more detailing while keeping the old pattern

But standard noise does not have any pattern. This defeats the purpose IMO.

Using Euler (not a) with blurrier settings makes it more obvious. I also decided to use Euler since its the only sampler that I have some small comprehension about and because it is not an ancestral sampler so it is solely using the injected noise rather than injecting its own.

Also it is important to note that the base model seems a lot worse at handling the entire workflow. Now I don't know why but I get a lot more upscaling artifacts and overall blurrier images than if I use a custom average merged model.

using bad settings to make things obvious

With perlin at upscale:

Without:

With:

Without:

As you can see it becomes more "smudged" with the usual noise since it is not matching what has been used initialy.

The nodes that I used and you might not have are here:

output latent size / batch size / print latent (also outputs noise distribution)

https://github.com/Extraltodeus/CustomComfyUINodes/blob/main/print_latent.py

VAE decode tiled with some more slider to make it faster (overlap at 8 and tile size at 128, make it 64 if you don't have enough memory)

https://github.com/Extraltodeus/CustomComfyUINodes/blob/main/VAEdecodeTiledOptions.py

The latent by ratio that I am using

https://github.com/Extraltodeus/CustomComfyUINodes/blob/main/LatentByRatio_not_mine.py

[Extraltodeus]

Bonus eye candy

[ntdviet]

@Extraltodeus

standard noise is more detailing while keeping the old pattern

But standard noise does not have any pattern. This defeats the purpose IMO.

And yet you get new details with a 20%-30% denoised sampling using another seed (or same seed but different noise)
I think you didn't inspect my workflow :( I dont interrupt the sampling mid-schedule anymore, and this is special for SDXL Base. Unlike you (step 0-15 of 25 steps), I do full schedule and get as much detail as possible out of 1st pass. If I interrupt mid-schedule and introduce not matching noise in 2nd pass, there will be smudges as you observed, as the sampler will try to bend the details halfway into another pattern. But since I finish the full schedule in 1st pass, in the 2nd pass I am free to introduce any kind of noise I want, and that is where I do a blend of Perlin with regular noise that suits the finish I'm looking for. My rule of thumb for this step is: perlin to create, regular to continuate. I can have my angels growing feathers all over the body if I crank up perlin, but if I just want more skin textures and everything else regulated, I will decrease perlin and increase regular noise.

Look here, this 2-pass image is okayish but I don't like the face skin, too flat and the eyes can have a bit more definition (btw this image was created with 100% perlin in 1st pass, 100% perlin + 33% regular in 2nd pass) :

So I will do another refiner round with SDXL base, Euler (u wanted Euler :P) and 15/60 steps to brush over the the pix.
If I do full Perlin for a refiner round, this is what I get, even at Perlin detail_level 0 (workflow embedded)

This kind of pic is perfect to spot the differences because you have soft edges and sharp edges, dark and bright spots, smooth gradients... to compare. You see the face is just got too much too heavy make up. The breast piece got too much high contrast details, too messed up. The soft feathers lost too much of it's fluffiness.
That's why I prefer a blend, where I can have more control. Here I do a mainly regular noise refiner with a touch of perlin. The face got more details, the feather keep a bit more fluffiness back, and the cloth is less messy. I can further tweak the perlin/regular noise ratio and still have cfg as a free lever for variations:

Take my workflow and try out the blending of noises. It's kind of fun. I think I got much more out of SDXL Base than expected. Result is near production ready by now. If you have a good custom merged SDXL model, please share. I'd like to compare but I bet the unbiased SDXL Base is much more versatile. So that's my motivation to optimize it's application.

[Extraltodeus]

I think you didn't inspect my workflow :(

I tried to import it but I missed multiple nodes so I couldn't. I did overall check your workflow despite being able to try it.

I will decrease perlin and increase regular noise.

I mean you're not wrong. Reinjecting the same noise pattern can also create artifacts which in turn will become weird details depending on the subject. This was more noticeable on the older/bad version that I had made as there would be garlands or flying feathers all over the place. Also I have seen seomwhere that indeed using multiple seeds can be useful to actually enhance the image. So that's great that you were able to use the whole process in a better and smarter way! :)

near production ready by now

May I ask what do you do that requires stable diffusion? I'm currently looking for a source of income and being able to monetize what I can do would be really handy.

[williamcboehmjr]

Also I bet a lot of production artists use stable Diffusion as a starting point for post production or story boarding for the start. It would be a better visual than sketches. You could even use roop to put the actors in the story boards! I haven't used roop yet. I have no idea if it's any good.

[williamcboehmjr]

Got them to load, placed them in the root directory of custom nodes.

[ntdviet]

@williamcboehmjr is it working for you now? If not give me a screenshot of the red flagged nodes and I can tell you which pack to install

[williamcboehmjr]

It's working but I had to get the nodes listed above and put them in the root nodes directory. Vae tiles I think was one of them. That would be great if these packs were in comfyui manager database. Also I wish that the manager had an auto update option for nodes

[ntdviet]

It takes a bit of extra effort to have our custom nodes included in the manager's list. I tried to do it once but it was skipped by the repo owner (probably not up to the specification standard) so, whatever :')) If you have come with us so far, you will get the nodes you need anyways :)

[ntdviet]

@Extraltodeus

I tried to import it but I missed multiple nodes so I couldn't. I did overall check your workflow despite being able to try it.

Sorry I didn't mention, it's basically your previous workflow just with the custom script's primitive https://github.com/pythongosssss/ComfyUI-Custom-Scripts

Reinjecting the same noise pattern can also create artifacts which in turn will become weird details depending on the subject.

Your latest version got the artifacts brushed out pretty nicely already. So this is more of an artistic choice what you want after a certain sampler pass. Introducing new noise for a resampling of 20-30% of steps can also help fixing errors, so it's all depending on the subjects. Changing the choice of model and sampler can push the result to a certain degree. Example: Euler is kind of soft brush, whileas dpmpp_2m_sde is more like hard pencil when you need more definition.

May I ask what do you do that requires stable diffusion?

Oh I'm doing side gigs for a photo studio and fashion designers. Not a big deal yet :) There are plenty of ideas what else to do with stable diffusion but none will earn serious money right away

(Edits: it's horrible to write something while a meeting is going on... so many typos and poorly written sentences :))

[Extraltodeus]

I don't know how to inject the correct amount of noise for img2img. I actually tried to do this on a custom sampler node that I had made for that purpose. An evolution of this node but gave up because I could just never find what worked. If you're using a ksampler advancer you can use the "start at step" parameter and start at something like 1 to 5 instead of 0 in order to alter the images for a relatively similar process as img2img but there is less control. That way should work.

[ntdviet]

@Extraltodeus:
Can't you just use the settings from my last workflow? It's pretty solid and I'm using the first 2 passes as a default for my photo-realistic workflows now.

Just cut away the adjustable parameters and set it fix to these step numbers because otherwise you would need to carefully alter the sigma calculation as well. An inexperienced user would only mess it up.

Would love to have this baked into just 1 node.

[ntdviet]

If you would still want to have the adjustable parameters, I'd recommend to add a 3rd pass and let the user adjust that

Who ever thought SDXL base could deliver such a detail rich photo. All 3 passes were sampled with just the base model.

[williamcboehmjr]

I like this workflow, but it can be slow. I am trying to find a great middle ground w/ this detail and structure...but somewhat fast...I have a 2070 Super w/ 8gb and I am not overclocking (whenever I do that it crashes SD of any sort, ha) but I find it funny that my favorite workflows come from a small github. but it's a great one @Extraltodeus and @city96 Love that you guys came together and made a great workflow baby. lol

[williamcboehmjr]

More than just a workflow, rethinking it all! Kinda like what Fooocus did but inside comfyui....you can adjust it to what you like, can't do that very much in Fooocus...but I do love that thing also. I might start going back to A1111 for some things now also.....maybe when they get their Controlnet stuff figured out...idk. but their adetailer chaining was perfect, I would do Person/Hands/Face Mesh/Eye Mesh/ and the output was just so great. I had a lot of win tonight...and I'm still drinking wine! I'm diffusin' drunk.

[Extraltodeus]

@ntdviet

Would love to have this baked into just 1 node.

I am putting most of this into one node the fun but it's a disgusting python mess lol

As for refiner coordination I have posted a node on civitai that does the job.

https://civitai.com/models/121394

I will check on your workflow later! :D

[ntdviet]

I got that sampler from Civit already. You didn't update it with your latest noise, no?

[Extraltodeus]

No I did not. I have to modify a few things before sharing it with you because it is currently a mess lol