Why does unlimited:waifu2x pad the entire image at once?

[LoganDark]

unlimited:waifu2x currently converts the entire image to one tensor, and pads it all at once. But my implementation of unlimited:waifu2x pads and edge bleeds for each tile individually:

https://github.com/LoganDark/waifu2x-unlimited/blob/bbdd2baa67a2ac3fec69fb76eb3c7b64097975eb/src/jobs/jobs.ts#L328-L422

https://github.com/LoganDark/waifu2x-unlimited/blob/bbdd2baa67a2ac3fec69fb76eb3c7b64097975eb/src/jobs/jobs.ts#L171-L191

Is there a reason why this isn't done in the official version?

[nagadomi]

Padding for neural network model inputs

In super-resolution model, it is better to avoid padding as much as possible.
The reason is that the pixels added by padding are fakes that are not part of the original image and cause the output borders to degraded.

There are no pixels outside of the entire image, so padding is the only solution for image borders(boundary).
However, pixels outside of each tile can exist, so it is better to use real image pixels instead of padding.

AlphaBorderPadding/EdgeBleed

My implementation of AlphaBorderPadding is a linear-time algorithm.
It means that the total processing time when processing each tile division and the processing time when processing the entire image at once are the same.
There is no particular advantage to tiling.
Also, when tiling, overlapping or intersecting tile borders and alpha borders can cause unstable results in some cases.

(Tiling for the neural network part is intended to avoid GPU(VRAM) out of memory. I am not concerned about that in AlphaBorderPadding.)

[LoganDark]

There are no pixels outside of the entire image, so padding is the only solution for image borders(boundary).
However, pixels outside of each tile can exist, so it is better to use real image pixels instead of padding.

This is not how my implementation works.

Here is a diagram:

This is done for each individual tile, instead of padding the whole image first.

[nagadomi]

Then they are practically the same thing.
I don't care much about code details anymore, as I sometimes write tens of thousands of lines of code alone.
I also think the unlimited:waifu2x code is not good. I developed it in a few days with javascript, which I haven't learned for decades. I'll try to improve it little by little :-)

At the moment, I am not too concerned about duplicate memory usage for images, but it will be necessary for more huge images to be processed.
It will need to be improved in the python version. I basically just ported the python version and didn't do any JavaScript-specific optimizations, unless it does not work.

[LoganDark]

Yeah in addition to my fork of unlimited:waifu2x I might write a Rust version too. It would use the updated algorithms that I learned from the web version, and probably outperform python

[nagadomi]

The most different part of the browser (CPU) and server (GPU) versions is that in the browser version, tiling helps to display visual progress.
With GPU, typical sized images are processed in less than a second, so users are not aware of tiling.
In the browser version, it takes several minutes to process, so without any visual feedback, users are unsure if it is working.

[nagadomi]

Yes, I just mentioned the differences in what users want in UI.
But it may make a difference in the limitations to achieve optimization in the end.

[LoganDark]

Wen you showed the blender example in other posts, you showed the padding was making the image almost twice its original size. So you don't end up with 2x memory usage, but 5x.

If this was done per tile, then you would only have to generate enough padding to fill a single tile at once.

I also use tiles that are not square (i.e. at bottom / right edge of image) which from my experience doesn't seem to result in any difference compared to using square tiles with very large padding.

Number 1 is not applicable to large images, because the tile size will always be much smaller than the image. But number 2 is applicable for medium sized images where the overhead of having tiles hang past the edge of the image is significant. Especially if those tiles are larger (as larger tiles are more efficient, due to less duplication of context, which is required for each tile).

[LoganDark]

I've successfully run a tile size of 1024 in the browser, btw. And also on my GPU (where that tile size takes less than 15ms).

I fully believe your metric that https://waifu2x.udp.jp/ can handle more than 1 frame per second.

[nagadomi]

When using GPU, increasing the batch size will be faster than increasing the tile size (but if the tile size is too small, it will be inefficient).
With nunif's waifu2x.cli, specify the following. You can see the number of images processed/second (this depends on the image size and GPU spec, of course).

python -m waifu2x.cli --style art -m scale -i ./tmp/input-dir/ -o ./tmp/output-dir/ --tile-size 256 --batch-size 16

100%|███████████████████████| 46/46 [00:14<00:00,  3.10it/s]

[LoganDark]

of course, I have started to experiment with batch size on the web version. I already use it for images with alpha channel, if TTA is off, the rgb and alpha will be batched together, and the model will only run once across both of them.

This doesn't seem to have much benefit on the CPU. But if one day WebGPU works (and you seem to be pushing for the WebGPU backend to support the waifu2x ONNX models, thank you!!), then batching may suddenly have a very large benefit. Because suddenly GPU's unused execution units (because model execution is still somewhat serial) can work on those tensors in parallel.

It's one of the reasons why I tried running loading models in parallel or running multiple models in parallel. But unfortunately ONNX for web does not properly support this. Trying to load multiple models in parallel just causes some of the promises to never resolve, and trying to run models in parallel just causes them to run in sequence.

But with WebGPU and batching, it should be possible to make the web version much much faster. Even faster than the improvement that WebGPU would already give.