Rendering stuck on cell that reads data #1428

Dr4cky · 2023-12-10T13:09:16Z

Hi there,

I am having an issue and I have no idea why it is not working. The rendering gets stuck on the following cell:

path = Path(r"data.csv")
df = pd.read_csv(path, index_col=0, delimiter=';')

It doesn't show an error, it just keeps loading forever "Executing 2 out of 6".
If I just simply run this cell in my notebook, it runs instantly. Any idea what is going wrong?

Thanks!

The text was updated successfully, but these errors were encountered:

ggogate · 2023-12-10T14:39:22Z

Anything in the browser Console?
Also can you try loading a different, smaller sample CSV which is comma separated.

Dr4cky · 2023-12-10T14:48:30Z

Anything in the browser Console? Also can you try loading a different, smaller sample CSV which is comma separated.

I just found out that it is not actually that cell that causes the problem, but the cell after:

returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

A portion of the error that I see in the console is as follows:

id
raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id)
tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: c5416394-05a1-485e-a9bb-595a9d5e2387)
404 GET /api/kernels/c5416394-05a1-485e-a9bb-595a9d5e2387?1702219619084 (::1) 4.17ms

ggogate · 2023-12-10T15:10:15Z

Try commenting out the figure part and see if the HTML displays okay. Then try moving the fig_ts part above the 'with'.

Dr4cky · 2023-12-10T15:47:29Z

I did a reinstall and the current error that Voila gives me is that it cannot find the module plotly, even though it is installed in my environment and I can the code in the notebook just fine. Is it calling another kernel?

ggogate · 2023-12-10T15:51:07Z

Try starting Jupyter/voila with --enable_nbextensions=True

Dr4cky · 2023-12-10T15:54:29Z

Gave me the same errors unfortunately, so plotly module not found and kernel does not exist.

trungleduc · 2023-12-10T16:42:30Z

Hi, could post the full reproduction code? It is easier for us if you follow the issue template.

ggogate · 2023-12-10T16:56:18Z

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors.
Maybe try installing previous stable versions of libraries.
If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

Dr4cky · 2023-12-10T17:02:37Z

My code is as follows:

import numpy as np
import pandas as pd
from pathlib import Path
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import plotly.graph_objects as go

#%%
def show_ts(df):
    fig = go.Figure()
    for col in df.columns:
        fig.add_trace(
            go.Scatter(
                name=col,
                mode="markers+lines",
                x=df.index,
                y=df[col],
                hovertemplate=("Return: %{y:.2%}<extra></extra>"),
            )
        )
    fig.update_layout(
        title={"text": "Returns", "y": 0.9, "x": 0.5, "xanchor": "center", "yanchor": "top"},
        font_family="Times New Roman",
        title_font_family="Times New Roman",
        legend_title_font_color="green",
    )
    fig.layout.yaxis.tickformat = ",.0%"
    fig.update_xaxes(title_font_family="Arial")
    fig.update_layout(
        # height=600,
        paper_bgcolor="rgba(0,0,0,0)",
        plot_bgcolor="rgba(0,0,0,0)",
        legend=dict(font_family="Courier New", font_color="white"),
    )
    fig.show()

#%%
data = np.random.random((10, 3))
df = pd.DataFrame(data, columns=['A', 'B', 'C'])


returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

tab = widgets.Tab([returns_plot])
tab.set_title(0, 'Data')
header_one = widgets.HTML(value="<h4>Task One</h4>", layout=widgets.Layout(width='300px'))
display(widgets.VBox([header_one, tab]))

It is stuck on 2/4. The message I see in console is:

[Voila] WARNING | Clearing invalid/expired login cookie username-localhost-8869
[Voila] WARNING | Notebook dashboard2.ipynb is not trusted
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Dr4cky · 2023-12-10T17:09:43Z

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors. Maybe try installing previous stable versions of libraries. If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

My requirements is as follows:

ipykernel==6.27.1
ipywidgets==8.1.1
jupyterlab==4.0.9
nbformat==5.9.2
numpy==1.26.2
openpyxl==3.1.2
pandas==2.1.4
plotly==5.18.0
pre-commit==3.5.0
voila==0.5.5

Running Python version 3.12.1

ggogate · 2023-12-10T18:24:47Z

I was able to reproduce the issue with the same Python and library versions.
I was also able to run voila by making following changes -

Removing the imports to interact - #from ipywidgets import interact, interactive, fixed, interact_manual
Change the fig.show() to return a fig. Alternately add the with-display in the show_ts()
#fig.show()
return fig
Add a display("") before the with and move the show_ts above the with
#%%
data = np.random.random((10, 3))
df = pd.DataFrame(data, columns=['A', 'B', 'C'])
display("Before")
fig_ts = show_ts(df)
#display("After")
returns_plot = widgets.Output()
with returns_plot:
text = widgets.HTML('Time series data.')
display(fig_ts)
#interact(fig_ts)
display(text)

Alternately, move the display part inside the show_ts() function.

Attaching both versions of notebooks -
Voila-Test.zip

Dr4cky · 2023-12-10T19:15:51Z

Thanks for your help! Unfortunately, Voila fails to run for both of your notebooks. It seems like display(fig_ts) also does not work.

Dr4cky · 2023-12-16T14:07:20Z

I tried Voila version 0.4.0, after which the code does work. Why would that be?

afonit · 2024-01-03T16:00:57Z

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

Dr4cky · 2024-01-03T16:36:37Z

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

Thanks! I gave up trying to figure it out and just used an older version of Voila (0.4.0). Hopefully they will fix it, though,

afonit · 2024-01-03T19:55:32Z

I have been trying to find a minimal reproducible example to aid in the troubleshooting/solution.

What adds to the difficulty - is it can be inconsistent, for example with the all current packages in a clean enviornment:

import ipywidgets as widgets
import pandas as pd
from IPython.display import display, HTML, Markdown, Javascript

data = {'a': [1, 2, 3],
        'b': ['one', 'two', 'three'],
       }

df = pd.DataFrame(data)

dropdown = widgets.Dropdown(
    description='select one',
    options=[1, 2, 3],
    value=1,
)

base_out = widgets.Output()

with base_out:
    base_out.clear_output()
    display(Markdown("""## This is in the output"""))
    display(df)

base_out

That will render fine every time.

but if I break it up into cells with some extra getting displayed eg:

Then it will frequently get stuck on:

and if I do a CTRL+SHIFT+r then the page will refresh and load the page completely.

With nothing in the logs other than:

Jan 03 14:51:08 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:53 test-env-149200 voila[18148]: [Voila] Kernel started: 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Debugger warning: It seems that frozen modules are being used, which may
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - to python to disable frozen modules.
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:56 test-env-149200 voila[18148]: [Voila] Connecting to kernel 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3.

trungleduc · 2024-01-03T20:13:10Z

Thanks for the reproduction, I will take a look at it.

afonit · 2024-01-17T18:18:29Z

@trungleduc, please let us/me know if there are any additional steps we could take to aid in nailing this down.

trungleduc · 2024-02-05T13:34:33Z

I'm able to reproduce the issue with a more compact snippet

import ipywidgets as widgets
import os
returns_plot = widgets.Output()
with returns_plot:
    display(widgets.HTML("Hello"))
    display(widgets.IntSlider())
returns_plot

with these dependencies

voila=0.5.5
ipywidgets=8.1.1
ipykernel=6.29.0
jupyter_client=8.6.0
nbclient=0.7.4

Voila will hang randomly after some refreshes. I think the issue is related to #1234 and #1253. Voila hangs because in this await call, nbclient does not receive the execute_reply message sent from the kernel side.

Moreover, from this comment (#1234 (comment)), it looks like this block is actually the culprit since if I remove it, Voila works as expected after multiple refreshes.

  if isinstance(stream, zmq.asyncio.Socket):
      assert stream is not None
      stream = zmq.Socket.shadow(stream.underlying)

Ping @davidbrochart @blink1073 if you have idea how to fix this.

blink1073 · 2024-02-06T00:12:03Z

Yes, the asyncio logic is definitely convoluted, I'm working on a fix in jupyter/jupyter_client#997, but I haven't had much bandwidth recently.

blink1073 · 2024-02-19T12:17:16Z

I don't think the issue is with that code block per se, because the effect is that it is not actually sending, because send_multipart returns an awaitable for an async socket.

ClaytonAstrom · 2024-04-02T19:54:29Z

Is there a better parent issue to track the discussions on this? Either on jupyter_server or something?

cosmicfarmers · 2024-04-04T05:42:14Z

For anyone who tries to resolve this issue.
In my case, preheating kernel solved problem, although it is not an ultimate solution; it didn't make the problem when preheating kernels.

ClaytonAstrom · 2024-04-15T15:56:50Z

I ended up having to back port some voila 0.5 changes to a forked version of 0.4.4, using jupyter-server 1.18.1. That worked for me.

afonit · 2024-06-07T19:51:17Z

Has anyone found a consistent workaround or solution for this? I know someone said the pre heated kernels worked for them as a workaround. We tried that and did not see a resolution to the issue.

trungleduc · 2024-06-07T20:41:22Z

@afonit do you have this issue without using the Output widget?

afonit · 2024-06-07T21:29:47Z

@trungleduc when I had just done a new vm (ubuntu 20.04) today and did everything fresh, it was the same issue as before, as you noted in:
#1428 (comment)

I am trying with these libraries:

voila=0.5.7
ipywidgets=8.1.3
ipykernel=6.29.4
jupyter_client=8.6.2
nbclient=0.10.0

We were using some very old packages that worked really well together but have not had to reimage and upgrade.

Right now it is being intermittent in working.

I did notice I am getting some messages like in: #1458

and

Jun 07 21:22:09 prod-705475 voila[61365]:     {%- block body -%}
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/share/jupyter/nbconvert/templates/base/null.j2", line 29, in block 'body'
Jun 07 21:22:09 prod-705475 voila[61365]:     {%- block body_loop -%}
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/share/jupyter/voila/templates/lab/index.html.j2", line 86, in block 'body_loop'
Jun 07 21:22:09 prod-705475 voila[61365]:     {% set page_config_full = page_config.copy() %}
Jun 07 21:22:09 prod-705475 voila[61365]:     ^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 07 21:22:09 prod-705475 voila[61365]:   File "/home/ubuntu/miniforge3/envs/data_analysis_lab/lib/python3.12/site-packages/jinja2/environment.py", line 487, in getattr
Jun 07 21:22:09 prod-705475 voila[61365]:     return getattr(obj, attribute)
Jun 07 21:22:09 prod-705475 voila[61365]:            ^^^^^^^^^^^^^^^^^^^^^^^
Jun 07 21:22:09 prod-705475 voila[61365]: jinja2.exceptions.UndefinedError: 'page_config' is undefined

To answer your output question, yes if I do some very basic notebooks with no outputs just markdown and a widget - they render fine in Voila.

trungleduc · 2024-06-07T21:35:18Z

I would avoid using the Output widget, and just render the normal widgets or group them in the HBox or VBox widget for now. Calling display inside an output context might be the root of the issue.

afonit · 2024-06-07T21:39:17Z

I would avoid using the Output widget, and just render the normal widgets or group them in the HBox or VBox widget for now. Calling display inside an output context might be the root of the issue.

We will try to do that - thank you. We have those heavily embedded id display's, we will try to simplify them and make it work - almost all our notebooks are widget/display heavy.

Thanks for your time and replies.

jvailius · 2024-06-12T09:01:21Z

I have the same issue as soon as I add a widget from ipywidgets (v8): Rendering progress bar is stuck a x out of y steps.

trungleduc · 2024-06-12T09:09:13Z

Hi @jvailius, could you provide your code to reproduce the issue? Does it involve using the Output widget?

jvailius · 2024-06-12T12:09:08Z

Hi @trungleduc - this is somewhat embarrassing because I can't reproduce it. I can not say why it works now with an interactive widget. Although from time to time it is stuck again - but rarely. Either is is a change to my code or some dependency that changed while trying different voila/widgets versions. Sorry for the noise. :/

trungleduc · 2024-06-12T12:20:02Z

@jvailius Actually this issue is quite hard to reproduce, it also happened to me, so I want to limit the possible culprits. The most possible reason is the Output widget.

jvailius · 2024-06-12T12:23:45Z

The most possible reason is the Output widget.

Forgot to mention that I was not using Output but it may have been related to fig.show() in plotly.

konosubakonoakua · 2024-08-28T09:30:54Z

same here

arthurBarthe · 2024-10-26T17:54:53Z

Hi,

Don't know if this might be helpful in solving the issue, but a fix for me has been to replace display (within out context voila was hanging) by append_display_data, as per the code below which I believe is similar.

anim = video_plot_3d(sample, get_title=lambda i: f'time: {i}', cmap='BrBG')
out = widgets.Output(layout={'border': '1px solid black'})
plt.close()
out.append_display_data(HTML(anim.to_html5_video()))
out

Apologies if this is unrelated.

trungleduc · 2024-10-28T10:26:31Z

Hi all, we released a new rendering method in Voila 0.5.8 (https://voila.readthedocs.io/en/stable/customize.html#changing-dashboard-rendering-technique). You can activate it by using the flag --progressive_rendering=True

Could you try this mode to see if the issue is still persistent?

Dr4cky added the bug Something isn't working label Dec 10, 2023

trungleduc self-assigned this Jan 3, 2024

afonit mentioned this issue Feb 7, 2024

Methods to troubleshoot Voila getting stuck on Executing x of y #1395

Open

Zsailer mentioned this issue Feb 15, 2024

Meeting Notes 2024 jupyter-server/team-compass#57

Open

blink1073 mentioned this issue Feb 18, 2024

Improve handling of async sockets jupyter/jupyter_client#1010

Closed

trungleduc mentioned this issue Feb 24, 2024

Browser continues heartbeat even though server has finished #1446

Closed

Rendering stuck on cell that reads data #1428

Rendering stuck on cell that reads data #1428

Comments

Dr4cky commented Dec 10, 2023

ggogate commented Dec 10, 2023 • edited Loading

Dr4cky commented Dec 10, 2023

ggogate commented Dec 10, 2023

Dr4cky commented Dec 10, 2023

ggogate commented Dec 10, 2023

Dr4cky commented Dec 10, 2023

trungleduc commented Dec 10, 2023 • edited Loading

ggogate commented Dec 10, 2023

Dr4cky commented Dec 10, 2023

Dr4cky commented Dec 10, 2023

ggogate commented Dec 10, 2023 • edited Loading

Dr4cky commented Dec 10, 2023

Dr4cky commented Dec 16, 2023

afonit commented Jan 3, 2024 • edited Loading

Dr4cky commented Jan 3, 2024

afonit commented Jan 3, 2024 • edited Loading

trungleduc commented Jan 3, 2024

afonit commented Jan 17, 2024

trungleduc commented Feb 5, 2024

blink1073 commented Feb 6, 2024

blink1073 commented Feb 19, 2024

ClaytonAstrom commented Apr 2, 2024

cosmicfarmers commented Apr 4, 2024

ClaytonAstrom commented Apr 15, 2024

afonit commented Jun 7, 2024 • edited Loading

trungleduc commented Jun 7, 2024

afonit commented Jun 7, 2024 • edited Loading

trungleduc commented Jun 7, 2024 • edited Loading

afonit commented Jun 7, 2024

jvailius commented Jun 12, 2024

trungleduc commented Jun 12, 2024

jvailius commented Jun 12, 2024

trungleduc commented Jun 12, 2024

jvailius commented Jun 12, 2024

konosubakonoakua commented Aug 28, 2024

arthurBarthe commented Oct 26, 2024

trungleduc commented Oct 28, 2024

ggogate commented Dec 10, 2023 •

edited

Loading

trungleduc commented Dec 10, 2023 •

edited

Loading

ggogate commented Dec 10, 2023 •

edited

Loading

afonit commented Jan 3, 2024 •

edited

Loading

afonit commented Jan 3, 2024 •

edited

Loading

afonit commented Jun 7, 2024 •

edited

Loading

afonit commented Jun 7, 2024 •

edited

Loading

trungleduc commented Jun 7, 2024 •

edited

Loading