Run poetry update and reinstall everything with pip

[hugobuddel]

This should prevent ScopeSim_Data from downgrading scopesim to 0.9.0 or whatever all the time.

[hugobuddel]

Draft because AstarVienna/ScopeSim#503 should first be merged, and then the slack bot should be turned on again. But conceptually this is ready.

[teutoburg]

Yeah, all the pip install -e <pkg> at the end are probably the best way to solve this now. That's what I have been doing locally anyway when testing if e.g. a change in ScopeSim would break an IRDB notebook. That is, have the local feature branch with that change checked out and running pip install -e, then run the notebooks.

Now, the question is, why do we still need the poetry commands in this script at all??

[hugobuddel]

Now, the question is, why do we still need the poetry commands in this script at all??

Because it is not possible to install the dependency groups without poetry..

See https://stackoverflow.com/questions/76118614/is-it-possible-to-install-poetry-groups-with-pip

But that page links to https://discuss.python.org/t/-/39233

and that links to https://peps.python.org/pep-0735/ "Dependency Groups in pyproject.toml", which is marked as Accepted, so maybe there will be a better way to get those dependency groups in soon

[teutoburg]

Now, the question is, why do we still need the poetry commands in this script at all??

Because it is not possible to install the dependency groups without poetry..

Ah right, I remember we've been there...

and that links to https://peps.python.org/pep-0735/ "Dependency Groups in pyproject.toml", which is marked as Accepted, so maybe there will be a better way to get those dependency groups in soon

Looks useful, ideally poetry would incorporate that once it's actually a thing, then we would get the best of both worlds...

[hugobuddel]

For some reason this is still

  - Installing nbclient (0.9.0)

and then

   - Downgrading nbclient (0.9.0 -> 0.5.13)

While this does not work. Version 0.10.0 of nbclient should work

[hugobuddel]

Apparently doing poetry install followed by poetry update gives a different (worse) result than doing poetry lock and then poetry install..

ScopeSim_Templates$ poetry shell
ScopeSim_Templates$ pip list | grep jupyter_client
# Nothing
ScopeSim_Templates$ poetry install --with=test,dev,docs
ScopeSim_Templates$ pip list | grep jupyter_client
jupyter_client                7.4.9
ScopeSim_Templates$ pip list | grep jupyter_client
jupyter_client                7.4.9

Oh FFS, rubber ducking by writing this comment, I realized I could try to add --with to poetry update, and indeed that works:

ScopeSim_Templates$ poetry update --with=test,dev,docs
ScopeSim_Templates$ pip list | grep jupyter_client
jupyter_client                8.6.3

But this --with is not required for poetry lock.

Maybe it's just me, but I find poetry so much more unintuitive than e.g. conda. Why would I only want to update the dependencies of certain groups?

[hugobuddel]

P.R. created to update the poetry documentation to save other people the trouble: python-poetry/poetry#9857

[hugobuddel]

Thinking more about this, we are still doing things wrong. Say speXtra has a weird test dependency, that is not required by ScopeSim, then that dependency would be removed when installing ScopeSim, thereby removing the ability to test speXtra.

Instead of installing all the packages, ScopeSim_Data should maybe be its own poetry project, and then this CI job should add all dependencies through

poetry add "git+https://github.com/AstarVienna/ScopeSim@main[dev,test,docs]"

(or maybe through the cloned local path with -E) and then install ScopeSim_Data. Or something like that.

Or we should finally move to ScopeSim_Core (the library) and have ScopeSim (the application) have all possible relevant dependencies as optional dependency groups, then it is possible to just install that. Hmm no that won't work either, because then say that one of the packages has a new dependency in the test group, but the package is not yet released itself, how to get that in.

[hugobuddel]

This should prevent the failure of the last nights because AstarVienna/ScopeSim#503 has been fixed on main, and this P.R. now actually allows ScopeSim_Data to test with the main branch of all our projects.

[teutoburg]

Instead of installing all the packages, ScopeSim_Data should maybe be its own poetry project

Yeah that came to my mind as well...

[teutoburg]

It's certainly better than it was before in terms of doing what we want this test to do.