Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nebari deployment #130

Open
jourdain opened this issue Oct 10, 2024 · 7 comments · Fixed by #132
Open

Nebari deployment #130

jourdain opened this issue Oct 10, 2024 · 7 comments · Fixed by #132

Comments

@jourdain
Copy link
Collaborator

The log seems to report the following

[I 241010 19:47:05 proxyhandlers:760] Process mainprocess failed with stdout: Some weights of the model checkpoint at facebook/detr-resnet-50 were not used when initializing DetrForObjectDetection: ['model.backbone
.conv_encoder.model.layer1.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tr
acked', 'model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked']
    - This IS expected if you are initializing DetrForObjectDetection from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model fro
m a BertForPreTraining model).
    - This IS NOT expected if you are initializing DetrForObjectDetection from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSeq
uenceClassification model).
    ERROR:asyncio:Task exception was never retrieved
    future: <Task finished name='Task-9' coro=<debounce.<locals>.decorator.<locals>.wrapper.<locals>.debounced() done, defined at /Users/aktech/miniconda3/envs/jhub-apps-dev/lib/python3.12/site-packages/nrtk_explor
er/library/debounce.py:17> exception=TypeError("object NoneType can't be used in 'await' expression")>
    Traceback (most recent call last):
      File "/Users/aktech/miniconda3/envs/jhub-apps-dev/lib/python3.12/site-packages/nrtk_explorer/library/debounce.py", line 22, in debounced
        await func(*args, **kwargs)
    TypeError: object NoneType can't be used in 'await' expression
====================================================
@jourdain
Copy link
Collaborator Author

@PaulHax @vicentebolea any idea on what could be happening?

@vicentebolea
Copy link
Collaborator

It looks like a libraries version issue, maybe moving to an older version of transforms could "fix" this

PaulHax added a commit to PaulHax/nrtk-explorer that referenced this issue Oct 10, 2024
@PaulHax
Copy link
Collaborator

PaulHax commented Oct 10, 2024

Not 100% sure this gets it, but I get the same error when I change debounce.py to use create_task from trame.app.asynchronous rather than from asyncio

https://github.com/Kitware/nrtk-explorer/pull/132/files#diff-d7955bbc308df5cde56ee6be7255b7d3bd6b4da234963e107f0ef9eea7b948c8L28-R

@jourdain
Copy link
Collaborator Author

trame.app.asynchronous attach a callback to be aware of a failure on the task that will be hidden if you just call asyncio.create_task

PaulHax added a commit to PaulHax/nrtk-explorer that referenced this issue Oct 11, 2024
PaulHax added a commit that referenced this issue Oct 11, 2024
@jourdain
Copy link
Collaborator Author

@PaulHax can you double check that is truly resolved?

It worked fine without the proxy and I suspected there is some issue with overlapping aysnc servers from proxy and your app, but
before digging too deep, I wanted to share it with you to find out if there is something app related you can fix.

Also, I don't see the error 100% of the time, sometimes it works (in the proxy) and sometimes it throws the above error.

The command I run to test quickly with proxy (CLI, outside of jhub-apps):

python -m jhsingle_native_proxy.main --authtype=none --no-force-alive --conda-env= python {-}m nrtk_explorer.app.main {--}server {--}host 0.0.0.0 {--}port {port}

@PaulHax
Copy link
Collaborator

PaulHax commented Oct 15, 2024

I'm not reproducing the error =/

We have not release the hypothedical fix 8841d82 to PyPI or Conda. Is Nebari pulling nrtk-explorer from main branch or using package manager?

I'm happy to remove the one place we are debouncing: https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/app/core.py#L102

Was not much of an UX improvment.

@PaulHax PaulHax reopened this Oct 15, 2024
@jourdain
Copy link
Collaborator Author

Once we have that fix in a release, I should try it on Nebari to see if we fixed the issue. Nebari is using a conda environment, but we do a pip install inside that env.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants