Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tutor Python 3.11 upgrade #1006

Closed
2 tasks done
DawoudSheraz opened this issue Feb 21, 2024 · 7 comments
Closed
2 tasks done

Tutor Python 3.11 upgrade #1006

DawoudSheraz opened this issue Feb 21, 2024 · 7 comments
Assignees

Comments

@DawoudSheraz
Copy link
Contributor

DawoudSheraz commented Feb 21, 2024

  • Update Python to 3.11 in Dockerfile for nightly and ensure the image builds as expected. The changes would be merged to nightly. This task would rely on upgrade of edx-platform to Python 3.11
  • Ensure tutor codebase is compatible with Python 3.12

See #1004 for the context.

@DawoudSheraz
Copy link
Contributor Author

DawoudSheraz commented Feb 29, 2024

CI checks, docs fixup -- #1008

@DawoudSheraz
Copy link
Contributor Author

Attempted building Open edX docker file on Python 3.12. I had to make the following changes /tutor/templates/build/openedx/Dockerfile

L27-L30 (Python and Pyenv)

ARG PYTHON_VERSION=3.12.2
ENV PYENV_ROOT /opt/pyenv
RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.36 --depth 1

L96 - L102 (setuptools and pip update)

RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
    pip install \
    # https://pypi.org/project/setuptools/
    # https://pypi.org/project/pip/
    # https://pypi.org/project/wheel/
    setuptools==69.1.1 pip==24.0 wheel==0.41.2

However, the build failed when installing Open edX base requirements with the error:

 Downloading numpy-1.22.4.zip (11.5 MB)
#40 115.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 1.6 MB/s eta 0:00:00
#40 116.3   Installing build dependencies: started
#40 124.2   Installing build dependencies: finished with status 'done'
#40 124.2   Getting requirements to build wheel: started
#40 124.4   Getting requirements to build wheel: finished with status 'error'
#40 124.4   error: subprocess-exited-with-error
#40 124.4   
#40 124.4   × Getting requirements to build wheel did not run successfully.
#40 124.4   │ exit code: 1
#40 124.4   ╰─> [33 lines of output]
#40 124.4       Traceback (most recent call last):
#40 124.4         File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#40 124.4           main()
#40 124.4         File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#40 124.4           json_out['return_val'] = hook(**hook_input['kwargs'])
#40 124.4                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4         File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
#40 124.4           backend = _build_backend()
#40 124.4                     ^^^^^^^^^^^^^^^^
#40 124.4         File "/openedx/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
#40 124.4           obj = import_module(mod_path)
#40 124.4                 ^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4         File "/opt/pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py", line 90, in import_module
#40 124.4           return _bootstrap._gcd_import(name[level:], package, level)
#40 124.4                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#40 124.4         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#40 124.4         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#40 124.4         File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
#40 124.4         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#40 124.4         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#40 124.4         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#40 124.4         File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#40 124.4         File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#40 124.4         File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#40 124.4         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#40 124.4         File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
#40 124.4           import setuptools.version
#40 124.4         File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
#40 124.4           import pkg_resources
#40 124.4         File "/tmp/pip-build-env-ejuvnl5g/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
#40 124.4           register_finder(pkgutil.ImpImporter, find_on_path)
#40 124.4                           ^^^^^^^^^^^^^^^^^^^
#40 124.4       AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
#40 124.4       [end of output]

At first, I thought this was due to setuptools version. It is sort-of due to setuptools but as an indirect dependency of numpy. The numpy version in quince is 1.22.4.
Numpy version less than 1.26.0 do not work out of box with Python 3.12 due to pinned setuptools. This has been fixed by numpy as part of numpy/numpy#23808. However, because the requirements in edx-platform base.txt are pinned, I suspect the docker image build will fail until the underlying code in platform is updated to 3.12.

@kdmccormick kdmccormick changed the title Tutor codebase Python 3.12 upgrade Tutor codebase Python 3.11 upgrade Mar 21, 2024
@kdmccormick kdmccormick changed the title Tutor codebase Python 3.11 upgrade Tutor Python 3.11 upgrade Mar 21, 2024
@regisb
Copy link
Contributor

regisb commented Mar 26, 2024

Yes, this issue is currently blocked by the pending edx-platform upgrade.

@DawoudSheraz
Copy link
Contributor Author

The upstream has a PR to upgrade edx-platform to Python 3.11 (openedx/edx-platform#34374).
I have created a draft PR to upgrade Dockerfile to use Python 3.11 by upgrading pyenv #1056.

Régis had suggested using bookworm image on tutor-discovery (overhangio/tutor-discovery#67 (comment)). In tutor-notes, the slim-bookworm image has been working as expected (https://github.com/overhangio/tutor-notes/pull/36/files). For tutor, I am testing python:3.11-slim-bookworm but I don't expect it to be a blocker in releasing 3.11 changes in Tutor.

@feanil
Copy link
Contributor

feanil commented May 2, 2024

FYI, I've just merged the edx-platform Python 3.11 upgrade PR so you should be all set to update the edx-platform Tutor images.

@DawoudSheraz DawoudSheraz moved this from Blocked to In Progress in Tutor project management May 2, 2024
@DawoudSheraz DawoudSheraz moved this from In Progress to In review in Tutor project management May 7, 2024
@DawoudSheraz DawoudSheraz moved this from In review to Done in Tutor project management May 9, 2024
@DawoudSheraz
Copy link
Contributor Author

Closing this issue. The slim-bookworm update will be handled in a follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

3 participants