image outputs stop working

[htoyryla]

Image outputs should be updated every 1 sec, but this stops working seemingly randomly, most likely after the UI has been inactive and then a new generation is started.

This is quite serious as one has all settings and inputs ready, yet cannot see the outputs, and reloading the page will throw all inputs and settings away.

Maybe having every on all the time is a bad idea, and we should create the every handler only when the generation starts? Or alternatively discard the every=1 handler and instead use diffusion_run as a generator yielding images which are then read in a loop and displayed.

It is not obvious however, how gradio handles the browser - server interaction. under the hood.

[htoyryla]

Commit 5d45c92 now handles image output updating with yield and not refreshing at every 1 sec.

This also means that the images stay local to each session. Full multiuser operation is still not supported, as the model is still global.

[htoyryla]

The problems still occurs, at least partly same as issue #8 . Currently, diffusion stops at an iteration without error message, GO button does not respond any more and the queue is released (leading to the impression that the operation was captured by another client).

Happens very often on iPad, also otherwise it seems not to happen on all devices.

Anyhow, using yield did not solve the problem, indicating that the iteration loop gets stuck on the client side.

[htoyryla]

I have been able to reproduce the problem using the following test script so as to show it is indeed a gradio issue and will report it to their repo.

import torch
from torchvision import transforms
import torchvision.transforms.functional as TF
from PIL import Image

import gradio as gr

import time
       
transform = transforms.Compose([transforms.Resize((768, 768)), transforms.ToTensor()]) 

tensor_to_pil = transforms.ToPILImage()

# diffusion run to be run by each user instance

def diffusion_run(progress):
    
    progress(0)

    # get initial noise (mixed with init image if present)
    
    for i in range(0,100):

        with torch.no_grad():
            x = torch.zeros(1,3,768,768).normal_(0,1)
            
            x -= x.min()
            x /= x.max()

            imout_raw = tensor_to_pil(x[0])  
                      
            imout = tensor_to_pil(1 - x[0])
            
            print(i)
            
            time.sleep(1)
            
            progress(float(i) / 100)
        
            yield imout, imout_raw, str(i)+"/"+str(100)
            
# gradio UI           
      
with gr.Blocks() as demo:
    
    imo = gr.State(Image.new('RGB', (768, 768)))
    
    with gr.Row():
        with gr.Column(min_width = 80):
            html = '<div style="font-size: 36px; color:#666666;">#urdiffusion</div><br>'+  \
            '<div style="font-size: 12px; color:#666666;">experimental single user version</div>' + \
            '<div style="font-size: 20px; color:#666666;">by @htoyryla 2023</div>'
            logo = gr.HTML(html)
        with gr.Column():
            text_input = gr.Textbox(label="Prompt")
            textw = gr.Slider(minimum=0., maximum=100., value=10., label="Text weight")
        with gr.Column():
            skip = gr.Slider(minimum=0, maximum=100, value=10, label="Skip")
            mul = gr.Slider(minimum=0., maximum=1., value=1., label="Noise level")
            weak = gr.Slider(minimum=0., maximum=1., value=0., label="Softness")
        init_image = gr.Image()
        with gr.Column():
            text_button = gr.Button("Go")
            process_status = gr.Textbox(label="Generation status")
        
    with gr.Row():    
        image_output_raw = gr.Image(label="generated")
        image_output = gr.Image(label="postprocessed")
    
    
    # function to run diffusion based on current settings    
    
    def run(im):
        
        process_status.value = "Processing..."
        for im, imr, i in diffusion_run(progress=gr.Progress()):
            #print(i)
            imo = imr
            yield im, imr, str(i)     
            
    # buttons to trigger diffusion and post processing        
             
    text_button.click(queue=True, fn=run, inputs=[init_image], outputs=[image_output, image_output_raw, process_status])
    

demo.queue(concurrency_count=1)
demo.launch(server_name = "0.0.0.0")

[htoyryla]

Filed an issue here gradio-app/gradio#4761