Use Python 3.11 by default and venv instead of virtualenv

[bgruening]

Closes usegalaxy-eu/issues#465.

[bgruening]

Please take this one over, I will be on a bike soon.

[sanjaysrikakulam]

Also, all the changes are to sn07 files and not sn06.

[kysrpex]

Also, all the changes are to sn07 files and not sn06.

Do you mean that the same changes should be applied to sn06's playbook and group vars or that this changeset is correct?

[sanjaysrikakulam]

Also, all the changes are to sn07 files and not sn06.

Do you mean that the same changes should be applied to sn06's playbook and group vars or that this changeset is correct?

We do not have a sn07 host at the moment. Only sn06. Also, the changes @bgruening made and suggested were concerning to sn06. Though the changes made to sn07 now are useful (keep things synced), they will not change or implement @bgruening's plan for sn06. Maybe that will come in a separate PR?

[kysrpex]

Also, all the changes are to sn07 files and not sn06.

Do you mean that the same changes should be applied to sn06's playbook and group vars or that this changeset is correct?

We do not have a sn07 host at the moment. Only sn06. Also, the changes @bgruening made and suggested were concerning to sn06. Though the changes made to sn07 now are useful (keep things synced), they will not change or implement @bgruening's plan for sn06. Maybe that will come in a separate PR?

Sorry I was thinking somehow of the maintenance node as sn07 (obviously wrong) 😅 Now it is clear.

Though the changes made to sn07 now are useful (keep things synced)

In the end what would make better sense is to define a servers group (e.g. "headnodes") and move most of the group vars currently defined for sn06 to such group.

[bgruening]

I could not find those variables in sn06. Sorry.

[sanjaysrikakulam]

I have added the missing variable now.

Notes
tiaas_virtualenv_python variable in sn06 and sn07 group_vars files is not required in latest version of the role (See here). So we can remove this variable when we migrate to the latest version of the TIaaS2 ansible role.

@bgruening's migration plan:

1. turn down handlers
2. mv /opt/galaxy/venv /opt/galaxy/venv39
3. recreate a new venv like above
4. run the playbook
5. sync to NFS for celery and others
6. turn down the manually started gunicorn

Updated plan:

1. turn down handlers
2. mv /opt/galaxy/venv /opt/galaxy/venv38
3. Merge this PR and run the playbook
4. sync to NFS for celery and others
5. turn down the manually started gunicorn and remove the venv311 and the gunicorn socket along with the systemd file (if created)

The changes will now use the miniconda role and create a miniconda directory in /opt/ where a _galaxy_ conda env with python 3.11 will be created. Python from this conda env will then create venv for Galaxy. Galaxy's venv will be created in /opt/galaxy/venv.

[sanjaysrikakulam]

It should work. Please merge this banner update PR and then this PR. Make sure to change the name of the existing venv directory to venv38 before running and deploying this playbook.

[bgruening]

I had workshops to do this morning and many questions, so will do it tomorrow morning, your midnight.

[sanjaysrikakulam]

@bgruening, tensorflow==2.9.3 is not available for Python 3.11. The conditional dependency should be updated to tensorflow==2.12.0. However, I am unsure whether the tool recommendation tool would work with tensorflow==2.12.0 version. This is the 1st version (latest version is 2.13.0) of Tensorflow that supports Python 3.11.

[bgruening]

Ok I will keep an eye on it. I guess we would deactivate it if it's not working and ask Anyp to look at it.

[bgruening]

And let's go!