diff --git a/CHANGELOG.md b/CHANGELOG.md index 34f5d467ea..1fb3cb5150 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,18 @@ instructions, because git commits are used to generate release notes: + +## v17.0.0 (2023-12-09) + +- 💥[Feature] Upgrade to Quince. (by @regisb) +- 💥[Feature] Replace "*.local.overhang.io" hostnames by "*.local.edly.io". (by @regisb) +- 💥[Feature] Enable the Indigo theme by default, if no other theme is set. (by @regisb) +- 💥[Deprecation] Tutor no longer supports the legacy Docker builder, which was previously available by setting `DOCKER_BUILDKIT=0` in the host environment. Going forward, Tutor will always use BuildKit (a.k.a. `docker buildx` in Docker v19-v22, or just `docker build` in Docker v23). This transition will improve build performance and should be seamless for Tutor users who are running a supported Docker version (by @kdmccormick). +- 💥[Deprecation] The template variable `is_buildkit_enabled`, which now always returns True, is deprecated. Plugin authors should assume BuildKit is enabled and remove the variable from their templates (by @kdmccormick). +- 💥[Deprecation] Adding Python packages to edx-platform via `private.txt` is no longer supported. Instead, users should bind-mount their requirement directories with `tutor mounts add ...`. (by @regisb) +- [Bugfix] Updated how the Tutor setting `JWT_RSA_PRIVATE_KEY` is rendered into the LMS Django setting `JWT_AUTH['JWT_PRIVATE_SIGNING_JWK']` as required by a recent breaking upstream change. The new representation of the `JWT_PRIVATE_SIGNING_JWK` simply adds the `dq`, `dp`, and `qi` parameters. Without this fix, LMS would encounter an `InvalidKeyError` on all logins. (by @kdmccormick) +- [Improvement] You don't have to run `tutor config save` every time you enable or disable a plugin anymore. (by @CodeWithEmad) + ## v16.1.8 (2023-12-10) diff --git a/docs/configuration.rst b/docs/configuration.rst index 2efd18054f..0f88d79e19 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -136,7 +136,7 @@ Open edX customisation This defines the git repository from which you install Open edX platform code. If you run an Open edX fork with custom patches, set this to your own git repository. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option. -- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/palm.4"``, or ``master`` in :ref:`nightly `) +- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/quince.1"``, or ``master`` in :ref:`nightly `) This defines the default version that will be pulled from all Open edX git repositories. @@ -156,9 +156,9 @@ These two configuration parameters define which Redis database to use for Open e .. _openedx_extra_pip_requirements: -- ``OPENEDX_EXTRA_PIP_REQUIREMENTS`` (default: ``["openedx-scorm-xblock>=16.0.0,<17.0.0"]``) +- ``OPENEDX_EXTRA_PIP_REQUIREMENTS`` (default: ``[]``) -This defines extra pip packages that are going to be installed for Open edX. +Define extra pip packages that are going to be installed for edx-platform. - ``NPM_REGISTRY`` (default: ``"https://registry.npmjs.org/"``) @@ -354,36 +354,16 @@ See :ref:`the corresponding tutorial `. .. _custom_extra_xblocks: -Installing extra xblocks and requirements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Would you like to include custom xblocks, or extra requirements to your Open edX platform? Additional requirements can be added to the ``OPENEDX_EXTRA_PIP_REQUIREMENTS`` parameter in the :ref:`config file ` or to the ``env/build/openedx/requirements/private.txt`` file. The difference between them, is that ``private.txt`` file, even though it could be used for both, :ref:`should be used for installing extra xblocks or requirements from private repositories `. For instance, to include the `polling xblock from Opencraft `_: +Would you like to include custom xblocks, or extra requirements to your Open edX platform? Additional requirements can be added to the ``OPENEDX_EXTRA_PIP_REQUIREMENTS`` parameter in the :ref:`config file `. For instance, to include the `polling xblock from Opencraft `_: tutor config save --append OPENEDX_EXTRA_PIP_REQUIREMENTS=git+https://github.com/open-craft/xblock-poll.git -Alternatively, add the dependency to ``private.txt``:: - - echo "git+https://github.com/open-craft/xblock-poll.git" >> "$(tutor config printroot)/env/build/openedx/requirements/private.txt" - Then, the ``openedx`` docker image must be rebuilt:: tutor images build openedx -.. _extra_private_xblocks: - -Installing extra requirements from private repositories -******************************************************* - -When installing extra xblock or requirements from private repositories, ``private.txt`` file should be used, because it allows installing dependencies without adding git credentials to the Docker image. By adding your git credentials to the Docker image, you're risking leaking your git credentials, if you were to publish (intentionally or unintentionally) the Docker image in a public place. - -To install xblocks from a private repository that requires authentication, you must first clone the repository inside the ``openedx/requirements`` folder on the host:: - - git clone git@github.com:me/myprivaterepo.git "$(tutor config printroot)/env/build/openedx/requirements/myprivaterepo" - -Then, declare your extra requirements with the ``-e`` flag in ``openedx/requirements/private.txt``:: - - echo "-e ./myprivaterepo" >> "$(tutor config printroot)/env/build/openedx/requirements/private.txt" - .. _edx_platform_fork: Running a fork of ``edx-platform`` @@ -401,16 +381,16 @@ Note that your edx-platform version must be a fork of the latest release **tag** If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular: -- Do not try to run a fork from an older (pre-Palm) version of edx-platform: this will simply not work. +- Do not try to run a fork from an older (pre-Quince) version of edx-platform: this will simply not work. - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail. -- Do not try to run a fork from the open-release/palm.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/palm.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/palm.master branch. +- Do not try to run a fork from the open-release/quince.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/quince.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/quince.master branch. .. _i18n: Adding custom translations ~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform `__ as well as those from `openedx-i18n `__. +If you are not running Open edX in English (``LANGUAGE_CODE`` default: ``"en"``), chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform `__ as well as those from `openedx-i18n `__. Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance:: @@ -431,9 +411,9 @@ Then, add a "django.po" file there that will contain your custom translations:: .. warning:: Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation. -The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language `__. +The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language `__. -If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well `__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory. +If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well `__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory. To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy:: diff --git a/docs/dev.rst b/docs/dev.rst index dd29b00c1b..6b1f1975bf 100644 --- a/docs/dev.rst +++ b/docs/dev.rst @@ -28,7 +28,7 @@ This will perform several tasks. It will: * build the "openedx-dev" Docker image, which is based on the "openedx" production image but is `specialized for developer usage`_ (eventually with your fork), * stop any existing locally-running Tutor containers, * disable HTTPS, -* set ``LMS_HOST`` to `local.overhang.io `_ (a convenience domain that simply `points at 127.0.0.1 `_), +* set ``LMS_HOST`` to `local.edly.io `_ (a convenience domain that simply `points at 127.0.0.1 `_), * prompt for a platform details (with suitable defaults), * start LMS, CMS, supporting services, and any plugged-in services, * ensure databases are created and migrated, and @@ -42,8 +42,8 @@ Additionally, when a local clone of edx-platform is bind-mounted, it will: Once setup is complete, the platform will be running in the background: -* LMS will be accessible at `http://local.overhang.io:8000 `_. -* CMS will be accessible at `http://studio.local.overhang.io:8001 `_. +* LMS will be accessible at `http://local.edly.io:8000 `_. +* CMS will be accessible at `http://studio.local.edly.io:8001 `_. * Plugged-in services should be accessible at their documented URLs. Now, use the ``tutor dev ...`` command-line interface to manage the development environment. Some common commands are described below. @@ -113,7 +113,7 @@ The ``openedx-dev`` Docker image is based on the same ``openedx`` image used by - The user that runs inside the container has the same UID as the user on the host, to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository). - Additional Python and system requirements are installed for convenient debugging: `ipython `__, `ipdb `__, vim, telnet. -- The edx-platform `development requirements `__ are installed. +- The edx-platform `development requirements `__ are installed. If you are using a custom ``openedx`` image, then you will need to rebuild ``openedx-dev`` every time you modify ``openedx``. To so, run:: diff --git a/docs/faq.rst b/docs/faq.rst index f4907d854c..6467ecc4ac 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -38,7 +38,7 @@ The `devstack `_ is meant for development o Is Tutor officially supported by edX? ------------------------------------- -Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation method for Open edX: see the `official installation instructions `__. +Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation method for Open edX: see the `official installation instructions `__. What features are missing from Tutor? ------------------------------------- diff --git a/docs/install.rst b/docs/install.rst index df4069261c..b058ddced5 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -12,7 +12,7 @@ Requirements * Architecture: Both AMD64 and ARM64 are supported. * Required software: - - `Docker `__: v20.10.15+ + - `Docker `__: v24.0.5+ (with BuildKit 0.11+) - `Docker Compose `__: v2.0.0+ .. warning:: @@ -123,11 +123,11 @@ Major Open edX releases are published twice a year, in June and December, by the 4. Test the new release in a sandboxed environment. 5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`). -The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Olive to Palm and rebuild some Docker images, run:: +The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Palm to Quince and rebuild some Docker images, run:: tutor config save tutor images build all # list the images that should be rebuilt here - tutor local upgrade --from=olive + tutor local upgrade --from=palm tutor local launch .. _autocomplete: diff --git a/docs/plugins/intro.rst b/docs/plugins/intro.rst index cb75f63660..57fd8d46ae 100644 --- a/docs/plugins/intro.rst +++ b/docs/plugins/intro.rst @@ -29,10 +29,6 @@ Enable/disable a plugin:: tutor plugins enable myplugin tutor plugins disable myplugin -After enabling or disabling a plugin, the environment should be re-generated with:: - - tutor config save - The full plugins CLI is described in the :ref:`reference documentation `. .. _existing_plugins: diff --git a/docs/plugins/v0/gettingstarted.rst b/docs/plugins/v0/gettingstarted.rst index 8fdaaf42a9..1f597e1946 100644 --- a/docs/plugins/v0/gettingstarted.rst +++ b/docs/plugins/v0/gettingstarted.rst @@ -41,10 +41,6 @@ You can then enable your newly-created plugin:: tutor plugins enable googleanalytics -Update your environment to apply changes from your plugin:: - - tutor config save - You should be able to view your changes in every LMS and CMS settings file:: grep -r googleanalytics "$(tutor config printroot)/env/apps/openedx/settings/" diff --git a/docs/quickstart.rst b/docs/quickstart.rst index cad6dd2077..58cf4d4e32 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -22,7 +22,7 @@ Yes :) This is what happens when you run ``tutor local launch``: 2. Configuration files are generated from templates. 3. Docker images are downloaded. 4. Docker containers are provisioned. -5. A full, production-ready Open edX platform (`Palm `__ release) is run with docker-compose. +5. A full, production-ready Open edX platform (`Quince `__ release) is run with docker-compose. The whole procedure should require less than 10 minutes, on a server with good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary. diff --git a/docs/reference/indexes.rst b/docs/reference/indexes.rst index db06fcfadb..e9c337ea33 100644 --- a/docs/reference/indexes.rst +++ b/docs/reference/indexes.rst @@ -7,10 +7,10 @@ Plugin indexes are a great way to have your plugins discovered by other users. P Index file paths ================ -A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Palm" release: +A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Quince" release: -- https://overhang.io/tutor/main/palm/plugins.yml -- ``/path/to/your/local/index/palm/plugins.yml`` +- https://overhang.io/tutor/main/quince/plugins.yml +- ``/path/to/your/local/index/quince/plugins.yml`` To add either indexes, run the ``tutor plugins index add`` command without the suffix. For instance:: @@ -106,9 +106,9 @@ Manage plugins in development Plugin developers and maintainers often want to install local versions of their plugins. They usually achieve this with ``pip install -e /path/to/tutor-plugin``. We can improve that workflow by creating an index for local plugins:: # Create the plugin index directory - mkdir -p ~/localindex/palm/ + mkdir -p ~/localindex/quince/ # Edit the index - vim ~/localindex/palm/plugins.yml + vim ~/localindex/quince/plugins.yml Add the following to the index:: diff --git a/docs/tutor.rst b/docs/tutor.rst index 4036ff8678..25ec6279da 100644 --- a/docs/tutor.rst +++ b/docs/tutor.rst @@ -93,6 +93,13 @@ An optional BRANCH suffix may be appended to the release name to indicate that e `Officially-supported plugins `__ follow the same versioning pattern. As a third-party plugin developer, you are encouraged to use the same pattern to make it immediately clear to your end-users which Open edX versions are supported. +In Tutor and its officially-supported plugins, certain features, API endpoints, and older depenency versions are periodically deprecated. Generally, warnings are added to the Changelogs and/or the command-line interface one major release before support for any behavior is removed. In order to keep track of pending removals in the source code, comments containing the string ``REMOVE-AFTER-VXX`` should be used, where ```` is the last major version that must support the behavior. For example:: + + # This has been replaced with SOME_NEW_HOOK (REMOVE-AFTER-V25). + SOME_OLD_HOOK = Filter() + +indicates that this filter definition can be removed as soon as Tutor v26.0.0. + .. _contributing: Contributing to Tutor diff --git a/docs/tutorials/edx-platform.rst b/docs/tutorials/edx-platform.rst index fde1ea9737..7d60e6b74d 100644 --- a/docs/tutorials/edx-platform.rst +++ b/docs/tutorials/edx-platform.rst @@ -69,7 +69,7 @@ Quite often, developers don't want to work on edx-platform directly, but on a de cd /my/workspace/edx-ora2 git clone https://github.com/openedx/edx-ora2 . -Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt `__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**:: +Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt `__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**:: git checkout diff --git a/docs/tutorials/plugin.rst b/docs/tutorials/plugin.rst index 55d91bdefa..6469f49f66 100644 --- a/docs/tutorials/plugin.rst +++ b/docs/tutorials/plugin.rst @@ -38,9 +38,9 @@ Our plugin is disabled, for now. To enable it, we run:: $ tutor plugins enable myplugin Plugin myplugin enabled Configuration saved to /home/yourusername/.local/share/tutor/config.yml - You should now re-generate your environment with `tutor config save`. + Environment generated in /home/yourusername/.local/share/tutor/env -At this point you could re-generate your environment with ``tutor config save``, but there would not be any change to your environment... because the plugin does not do anything. So let's get started and make some changes. +At this point your environment was updated, but there would not be any change there... because the plugin does not do anything. So let's get started and make some changes. Modifying existing files with patches ------------------------------------- diff --git a/docs/tutorials/theming.rst b/docs/tutorials/theming.rst index 224a5595a2..6c98384612 100644 --- a/docs/tutorials/theming.rst +++ b/docs/tutorials/theming.rst @@ -46,7 +46,7 @@ Then, run a local webserver:: tutor dev start lms -The LMS can then be accessed at http://local.overhang.io:8000. You will then have to :ref:`enable that theme `:: +The LMS can then be accessed at http://local.edly.io:8000. You will then have to :ref:`enable that theme `:: tutor dev do settheme mythemename @@ -54,4 +54,4 @@ Watch the themes folders for changes (in a different terminal):: tutor dev run watchthemes -Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.overhang.io:8000. +Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.edly.io:8000. diff --git a/requirements/base.txt b/requirements/base.txt index e79b5bdcc2..53ea28465e 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -6,27 +6,27 @@ # appdirs==1.4.4 # via -r requirements/base.in -cachetools==5.3.1 +cachetools==5.3.2 # via google-auth certifi==2023.7.22 # via # kubernetes # requests -charset-normalizer==3.2.0 +charset-normalizer==3.3.1 # via requests click==8.1.7 # via -r requirements/base.in -google-auth==2.22.0 +google-auth==2.23.3 # via kubernetes idna==3.4 # via requests jinja2==3.1.2 # via -r requirements/base.in -kubernetes==27.2.0 +kubernetes==28.1.0 # via -r requirements/base.in markupsafe==2.1.3 # via jinja2 -mypy==1.5.1 +mypy==1.6.1 # via -r requirements/base.in mypy-extensions==1.0.0 # via mypy @@ -40,7 +40,7 @@ pyasn1==0.5.0 # rsa pyasn1-modules==0.3.0 # via google-auth -pycryptodome==3.18.0 +pycryptodome==3.19.0 # via -r requirements/base.in python-dateutil==2.8.2 # via kubernetes @@ -58,19 +58,17 @@ rsa==4.9 # via google-auth six==1.16.0 # via - # google-auth # kubernetes # python-dateutil tomli==2.0.1 # via mypy -typing-extensions==4.7.1 +typing-extensions==4.8.0 # via # -r requirements/base.in # mypy urllib3==1.26.18 # via - # google-auth # kubernetes # requests -websocket-client==1.6.2 +websocket-client==1.6.4 # via kubernetes diff --git a/requirements/dev.in b/requirements/dev.in index c53771c419..ef0d7ada7b 100644 --- a/requirements/dev.in +++ b/requirements/dev.in @@ -7,6 +7,12 @@ pyinstaller scriv twine +# Constraints +# for compatibility with sphinx-rtd-theme +# drop this constraint once sphinx-rtd-theme 2.0.0 comes out +# https://github.com/readthedocs/sphinx_rtd_theme/tags +docutils<0.19.0 + # Types packages types-docutils types-PyYAML diff --git a/requirements/dev.txt b/requirements/dev.txt index 8d8b265839..fd7ba3e747 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -4,21 +4,19 @@ # # pip-compile requirements/dev.in # -altgraph==0.17.3 +altgraph==0.17.4 # via pyinstaller appdirs==1.4.4 # via -r requirements/base.txt -astroid==2.15.6 +astroid==3.0.1 # via pylint attrs==23.1.0 # via scriv -black==23.7.0 +black==23.10.1 # via -r requirements/dev.in -bleach==6.0.0 - # via readme-renderer -build==0.10.0 +build==1.0.3 # via pip-tools -cachetools==5.3.1 +cachetools==5.3.2 # via # -r requirements/base.txt # google-auth @@ -27,9 +25,9 @@ certifi==2023.7.22 # -r requirements/base.txt # kubernetes # requests -cffi==1.15.1 +cffi==1.16.0 # via cryptography -charset-normalizer==3.2.0 +charset-normalizer==3.3.1 # via # -r requirements/base.txt # requests @@ -42,15 +40,17 @@ click==8.1.7 # scriv click-log==0.4.0 # via scriv -coverage==7.3.0 +coverage==7.3.2 # via -r requirements/dev.in cryptography==41.0.7 # via secretstorage dill==0.3.7 # via pylint -docutils==0.20.1 - # via readme-renderer -google-auth==2.22.0 +docutils==0.18.1 + # via + # -r requirements/dev.in + # readme-renderer +google-auth==2.23.3 # via # -r requirements/base.txt # kubernetes @@ -60,7 +60,9 @@ idna==3.4 # requests importlib-metadata==6.8.0 # via + # build # keyring + # pyinstaller # twine importlib-resources==6.1.1 # via keyring @@ -78,12 +80,12 @@ jinja2==3.1.2 # scriv keyring==24.2.0 # via twine -kubernetes==27.2.0 +kubernetes==28.1.0 # via -r requirements/base.txt -lazy-object-proxy==1.9.0 - # via astroid markdown-it-py==3.0.0 - # via rich + # via + # rich + # scriv markupsafe==2.1.3 # via # -r requirements/base.txt @@ -94,29 +96,32 @@ mdurl==0.1.2 # via markdown-it-py more-itertools==10.1.0 # via jaraco-classes -mypy==1.5.1 +mypy==1.6.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via # -r requirements/base.txt # black # mypy +nh3==0.2.14 + # via readme-renderer oauthlib==3.2.2 # via # -r requirements/base.txt # kubernetes # requests-oauthlib -packaging==23.1 +packaging==23.2 # via # black # build + # pyinstaller pathspec==0.11.2 # via black pip-tools==7.3.0 # via -r requirements/dev.in pkginfo==1.9.6 # via twine -platformdirs==3.10.0 +platformdirs==3.11.0 # via # black # pylint @@ -131,17 +136,17 @@ pyasn1-modules==0.3.0 # google-auth pycparser==2.21 # via cffi -pycryptodome==3.18.0 +pycryptodome==3.19.0 # via -r requirements/base.txt pygments==2.16.1 # via # readme-renderer # rich -pyinstaller==5.13.1 +pyinstaller==6.1.0 # via -r requirements/dev.in -pyinstaller-hooks-contrib==2023.7 +pyinstaller-hooks-contrib==2023.10 # via pyinstaller -pylint==2.17.5 +pylint==3.0.2 # via -r requirements/dev.in pyproject-hooks==1.0.0 # via build @@ -153,7 +158,7 @@ pyyaml==6.0.1 # via # -r requirements/base.txt # kubernetes -readme-renderer==41.0 +readme-renderer==42.0 # via twine requests==2.31.0 # via @@ -171,21 +176,19 @@ requests-toolbelt==1.0.0 # via twine rfc3986==2.0.0 # via twine -rich==13.5.2 +rich==13.6.0 # via twine rsa==4.9 # via # -r requirements/base.txt # google-auth -scriv==1.3.1 +scriv==1.5.0 # via -r requirements/dev.in secretstorage==3.3.3 # via keyring six==1.16.0 # via # -r requirements/base.txt - # bleach - # google-auth # kubernetes # python-dateutil tomli==2.0.1 @@ -203,11 +206,11 @@ twine==4.0.2 # via -r requirements/dev.in types-docutils==0.20.0.3 # via -r requirements/dev.in -types-pyyaml==6.0.12.11 +types-pyyaml==6.0.12.12 # via -r requirements/dev.in -types-setuptools==68.1.0.0 +types-setuptools==68.2.0.0 # via -r requirements/dev.in -typing-extensions==4.7.1 +typing-extensions==4.8.0 # via # -r requirements/base.txt # astroid @@ -218,21 +221,16 @@ typing-extensions==4.7.1 urllib3==1.26.18 # via # -r requirements/base.txt - # google-auth # kubernetes # requests # twine -webencodings==0.5.1 - # via bleach -websocket-client==1.6.2 +websocket-client==1.6.4 # via # -r requirements/base.txt # kubernetes wheel==0.41.2 # via pip-tools -wrapt==1.15.0 - # via astroid -zipp==3.16.2 +zipp==3.17.0 # via # importlib-metadata # importlib-resources diff --git a/requirements/docs.txt b/requirements/docs.txt index 80bb030110..c4f5c6e9c2 100644 --- a/requirements/docs.txt +++ b/requirements/docs.txt @@ -8,9 +8,9 @@ alabaster==0.7.13 # via sphinx appdirs==1.4.4 # via -r requirements/base.txt -babel==2.12.1 +babel==2.13.1 # via sphinx -cachetools==5.3.1 +cachetools==5.3.2 # via # -r requirements/base.txt # google-auth @@ -19,7 +19,7 @@ certifi==2023.7.22 # -r requirements/base.txt # kubernetes # requests -charset-normalizer==3.2.0 +charset-normalizer==3.3.1 # via # -r requirements/base.txt # requests @@ -32,7 +32,7 @@ docutils==0.18.1 # sphinx # sphinx-click # sphinx-rtd-theme -google-auth==2.22.0 +google-auth==2.23.3 # via # -r requirements/base.txt # kubernetes @@ -48,13 +48,13 @@ jinja2==3.1.2 # via # -r requirements/base.txt # sphinx -kubernetes==27.2.0 +kubernetes==28.1.0 # via -r requirements/base.txt markupsafe==2.1.3 # via # -r requirements/base.txt # jinja2 -mypy==1.5.1 +mypy==1.6.1 # via -r requirements/base.txt mypy-extensions==1.0.0 # via @@ -65,7 +65,7 @@ oauthlib==3.2.2 # -r requirements/base.txt # kubernetes # requests-oauthlib -packaging==23.1 +packaging==23.2 # via sphinx pyasn1==0.5.0 # via @@ -76,7 +76,7 @@ pyasn1-modules==0.3.0 # via # -r requirements/base.txt # google-auth -pycryptodome==3.18.0 +pycryptodome==3.19.0 # via -r requirements/base.txt pygments==2.16.1 # via sphinx @@ -107,7 +107,6 @@ rsa==4.9 six==1.16.0 # via # -r requirements/base.txt - # google-auth # kubernetes # python-dateutil snowballstemmer==2.2.0 @@ -140,17 +139,16 @@ tomli==2.0.1 # via # -r requirements/base.txt # mypy -typing-extensions==4.7.1 +typing-extensions==4.8.0 # via # -r requirements/base.txt # mypy urllib3==1.26.18 # via # -r requirements/base.txt - # google-auth # kubernetes # requests -websocket-client==1.6.2 +websocket-client==1.6.4 # via # -r requirements/base.txt # kubernetes diff --git a/requirements/plugins.txt b/requirements/plugins.txt index e67affd113..c78ef5773f 100644 --- a/requirements/plugins.txt +++ b/requirements/plugins.txt @@ -1,12 +1,13 @@ -# change version ranges when upgrading from palm -tutor-android>=16.0.0,<17.0.0 -tutor-cairn>=16.0.0,<17.0.0 -tutor-discovery>=16.0.0,<17.0.0 -tutor-ecommerce>=16.0.0,<17.0.0 -tutor-forum>=16.0.0,<17.0.0 -tutor-license>=16.0.0,<17.0.0 -tutor-mfe>=16.0.0,<17.0.0 -tutor-minio>=16.0.0,<17.0.0 -tutor-notes>=16.0.0,<17.0.0 -tutor-webui>=16.0.0,<17.0.0 -tutor-xqueue>=16.0.0,<17.0.0 +# change version ranges when upgrading from quince +tutor-android>=17.0.0,<18.0.0 +tutor-cairn>=17.0.0,<18.0.0 +tutor-credentials>=17.0.0,<18.0.0 +tutor-discovery>=17.0.0,<18.0.0 +tutor-ecommerce>=17.0.0,<18.0.0 +tutor-forum>=17.0.0,<18.0.0 +tutor-jupyter>=17.0.0,<18.0.0 +tutor-mfe>=17.0.0,<18.0.0 +tutor-minio>=17.0.0,<18.0.0 +tutor-notes>=17.0.0,<18.0.0 +tutor-webui>=17.0.0,<18.0.0 +tutor-xqueue>=17.0.0,<18.0.0 diff --git a/tests/commands/test_config.py b/tests/commands/test_config.py index 5c9dcea307..58b15edd93 100644 --- a/tests/commands/test_config.py +++ b/tests/commands/test_config.py @@ -94,9 +94,8 @@ def test_config_append_with_existing_default(self) -> None: ) config = tutor_config.load(root) assert isinstance(config["OPENEDX_EXTRA_PIP_REQUIREMENTS"], list) - self.assertEqual(2, len(config["OPENEDX_EXTRA_PIP_REQUIREMENTS"])) self.assertEqual( - "my-package==1.0.0", config["OPENEDX_EXTRA_PIP_REQUIREMENTS"][1] + ["my-package==1.0.0"], config["OPENEDX_EXTRA_PIP_REQUIREMENTS"] ) diff --git a/tests/commands/test_images.py b/tests/commands/test_images.py index 7b0957790d..32ddba885f 100644 --- a/tests/commands/test_images.py +++ b/tests/commands/test_images.py @@ -128,10 +128,8 @@ def test_images_build_plugin_with_args(self, image_build: Mock) -> None: "service1", ] with temporary_root() as root: - utils.is_buildkit_enabled.cache_clear() - with patch.object(utils, "is_buildkit_enabled", return_value=False): - self.invoke_in_root(root, ["config", "save"]) - result = self.invoke_in_root(root, build_args) + self.invoke_in_root(root, ["config", "save"]) + result = self.invoke_in_root(root, build_args) self.assertIsNone(result.exception) self.assertEqual(0, result.exit_code) image_build.assert_called() @@ -146,7 +144,9 @@ def test_images_build_plugin_with_args(self, image_build: Mock) -> None: "host", "--target", "target", + "--output=type=docker", "docker_args", + "--cache-from=type=registry,ref=service1:1.0.0-cache", ], list(image_build.call_args[0][1:]), ) diff --git a/tests/test_env.py b/tests/test_env.py index a79060991a..27cff9678b 100644 --- a/tests/test_env.py +++ b/tests/test_env.py @@ -260,7 +260,7 @@ def test_current_version_in_latest_env(self) -> None: ) as f: f.write(__version__) self.assertEqual(__version__, env.current_version(root)) - self.assertEqual("palm", env.get_env_release(root)) + self.assertEqual("quince", env.get_env_release(root)) self.assertIsNone(env.should_upgrade_from_release(root)) self.assertTrue(env.is_up_to_date(root)) diff --git a/tests/test_utils.py b/tests/test_utils.py index 21af475d3b..5d5a6428ee 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,9 +1,9 @@ import base64 import os +import subprocess import tempfile import unittest from io import StringIO -import subprocess from typing import List, Tuple from unittest.mock import MagicMock, mock_open, patch diff --git a/tutor/__about__.py b/tutor/__about__.py index 0ca5b63da5..33297dd3a4 100644 --- a/tutor/__about__.py +++ b/tutor/__about__.py @@ -2,7 +2,7 @@ # Increment this version number to trigger a new release. See # docs/tutor.html#versioning for information on the versioning scheme. -__version__ = "16.1.8" +__version__ = "17.0.0" # The version suffix will be appended to the actual version, separated by a # dash. Use this suffix to differentiate between the actual released version and diff --git a/tutor/commands/images.py b/tutor/commands/images.py index 059c7bd377..ed3174af63 100644 --- a/tutor/commands/images.py +++ b/tutor/commands/images.py @@ -156,7 +156,7 @@ def images_command() -> None: # Export image to docker. This is necessary to make the image available to docker-compose. # The `--load` option is a shorthand for `--output=type=docker`. default="type=docker", - help="Same as `docker build --output=...`. This option will only be used when BuildKit is enabled.", + help="Same as `docker build --output=...`.", ) @click.option( "-a", @@ -211,7 +211,7 @@ def build( command_args += ["--add-host", add_host] if target: command_args += ["--target", target] - if utils.is_buildkit_enabled() and docker_output: + if docker_output: command_args.append(f"--output={docker_output}") if docker_args: command_args += docker_args @@ -223,27 +223,19 @@ def build( image_build_args = [*command_args, *custom_args] # Registry cache - if utils.is_buildkit_enabled(): - if not no_registry_cache: - image_build_args.append( - f"--cache-from=type=registry,ref={tag}-cache" - ) - if cache_to_registry: - image_build_args.append( - f"--cache-to=type=registry,mode=max,ref={tag}-cache" - ) + if not no_registry_cache: + image_build_args.append(f"--cache-from=type=registry,ref={tag}-cache") + if cache_to_registry: + image_build_args.append( + f"--cache-to=type=registry,mode=max,ref={tag}-cache" + ) # Build contexts for host_path, stage_name in build_contexts.get(name, []): - if utils.is_buildkit_enabled(): - fmt.echo_info( - f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" - ) - image_build_args.append(f"--build-context={stage_name}={host_path}") - else: - fmt.echo_alert( - f"Unable to add {host_path} to the build context '{stage_name}' of image '{host_path}' because BuildKit is disabled." - ) + fmt.echo_info( + f"Adding {host_path} to the build context '{stage_name}' of image '{image}'" + ) + image_build_args.append(f"--build-context={stage_name}={host_path}") # Build images.build( diff --git a/tutor/commands/plugins.py b/tutor/commands/plugins.py index e80b6fc33e..45d8bd677f 100644 --- a/tutor/commands/plugins.py +++ b/tutor/commands/plugins.py @@ -1,6 +1,7 @@ from __future__ import annotations import os +import sys import tempfile import typing as t @@ -9,6 +10,7 @@ from tutor import config as tutor_config from tutor import exceptions, fmt, hooks, plugins, utils +from tutor.commands.config import save as config_save_command from tutor.plugins import indexes from tutor.plugins.base import PLUGINS_ROOT, PLUGINS_ROOT_ENV_VAR_NAME from tutor.types import Config @@ -133,17 +135,15 @@ def list_command(show_enabled_only: bool) -> None: @click.command(help="Enable a plugin") @click.argument("plugin_names", metavar="plugin", nargs=-1, type=PluginName()) -@click.pass_obj -def enable(context: Context, plugin_names: list[str]) -> None: - config = tutor_config.load_minimal(context.root) +@click.pass_context +def enable(context: click.Context, plugin_names: list[str]) -> None: + config = tutor_config.load_minimal(context.obj.root) for plugin in plugin_names: plugins.load(plugin) fmt.echo_info(f"Plugin {plugin} enabled") tutor_config.save_enabled_plugins(config) - tutor_config.save_config_file(context.root, config) - fmt.echo_info( - "You should now re-generate your environment with `tutor config save`." - ) + tutor_config.save_config_file(context.obj.root, config) + context.invoke(config_save_command, env_only=True) @click.command( @@ -153,22 +153,20 @@ def enable(context: Context, plugin_names: list[str]) -> None: @click.argument( "plugin_names", metavar="plugin", nargs=-1, type=PluginName(allow_all=True) ) -@click.pass_obj -def disable(context: Context, plugin_names: list[str]) -> None: - config = tutor_config.load_minimal(context.root) +@click.pass_context +def disable(context: click.Context, plugin_names: list[str]) -> None: + config = tutor_config.load_minimal(context.obj.root) disable_all = "all" in plugin_names disabled: list[str] = [] for plugin in tutor_config.get_enabled_plugins(config): if disable_all or plugin in plugin_names: fmt.echo_info(f"Disabling plugin {plugin}...") - hooks.Actions.PLUGIN_UNLOADED.do(plugin, context.root, config) + hooks.Actions.PLUGIN_UNLOADED.do(plugin, context.obj.root, config) disabled.append(plugin) fmt.echo_info(f"Plugin {plugin} disabled") if disabled: - tutor_config.save_config_file(context.root, config) - fmt.echo_info( - "You should now re-generate your environment with `tutor config save`." - ) + tutor_config.save_config_file(context.obj.root, config) + context.invoke(config_save_command, env_only=True) @click.command(name="update") diff --git a/tutor/commands/upgrade/__init__.py b/tutor/commands/upgrade/__init__.py index 9c1d203424..b7a4e028de 100644 --- a/tutor/commands/upgrade/__init__.py +++ b/tutor/commands/upgrade/__init__.py @@ -1,4 +1,4 @@ -# Note: don't forget to change this when we upgrade from palm +# Note: don't forget to change this when we upgrade from quince OPENEDX_RELEASE_NAMES = [ "ironwood", "juniper", @@ -7,4 +7,6 @@ "maple", "nutmeg", "olive", + "palm", + "quince", ] diff --git a/tutor/commands/upgrade/compose.py b/tutor/commands/upgrade/compose.py index ea0eaaf9dd..7aef353bda 100644 --- a/tutor/commands/upgrade/compose.py +++ b/tutor/commands/upgrade/compose.py @@ -43,6 +43,9 @@ def upgrade_from(context: click.Context, from_release: str) -> None: upgrade_from_olive(context, config) running_release = "palm" + if running_release == "palm": + running_release = "quince" + def upgrade_from_ironwood(context: click.Context, config: Config) -> None: click.echo(fmt.title("Upgrading from Ironwood")) diff --git a/tutor/commands/upgrade/k8s.py b/tutor/commands/upgrade/k8s.py index b84ac4f300..64cebaf354 100644 --- a/tutor/commands/upgrade/k8s.py +++ b/tutor/commands/upgrade/k8s.py @@ -42,6 +42,9 @@ def upgrade_from(context: click.Context, from_release: str) -> None: upgrade_from_olive(context.obj, config) running_release = "palm" + if running_release == "palm": + running_release = "quince" + def upgrade_from_ironwood(config: Config) -> None: upgrade_mongodb(config, "3.4.24", "3.4") diff --git a/tutor/config.py b/tutor/config.py index 2a0ee73731..b5f48f30c3 100644 --- a/tutor/config.py +++ b/tutor/config.py @@ -1,8 +1,8 @@ from __future__ import annotations -from copy import deepcopy -import typing as t import os +import typing as t +from copy import deepcopy from tutor import env, exceptions, fmt, hooks, plugins, serialize, utils from tutor.types import Config, ConfigValue, cast_config, get_typed @@ -153,18 +153,6 @@ def _load_config_defaults_yml( return items -@hooks.Filters.CONFIG_DEFAULTS.add() -def _set_openedx_common_version_in_nightly( - items: list[tuple[str, t.Any]] -) -> list[tuple[str, t.Any]]: - # REMOVE-AFTER-v16 move this callback to the dedicated openedx/plugin.py module - from tutor.__about__ import __version_suffix__ - - if __version_suffix__ == "nightly": - items.append(("OPENEDX_COMMON_VERSION", "master")) - return items - - def get_template(filename: str) -> Config: """ Get one of the configuration templates. @@ -328,7 +316,9 @@ def _enable_plugins(root: str) -> None: enable_plugins(config) -@hooks.Actions.PLUGIN_UNLOADED.add() +# This is run with a very high priority such that it is called before the plugin hooks +# are actually cleared. +@hooks.Actions.PLUGIN_UNLOADED.add(priority=hooks.priorities.HIGH - 1) def _remove_plugin_config_overrides_on_unload( plugin: str, _root: str, config: Config ) -> None: @@ -342,7 +332,7 @@ def _remove_plugin_config_overrides_on_unload( fmt.echo_info(f" config - removing entry: {key}={value}") -@hooks.Actions.PLUGIN_UNLOADED.add(priority=100) +@hooks.Actions.PLUGIN_UNLOADED.add(priority=hooks.priorities.LOW) def _update_enabled_plugins_on_unload(_plugin: str, _root: str, config: Config) -> None: """ Update the list of enabled plugins. diff --git a/tutor/env.py b/tutor/env.py index 08db87dc6d..cccce94d0d 100644 --- a/tutor/env.py +++ b/tutor/env.py @@ -54,7 +54,6 @@ def _prepare_environment() -> None: ("HOST_USER_ID", utils.get_user_id()), ("TUTOR_APP", __app__.replace("-", "_")), ("TUTOR_VERSION", __version__), - ("is_buildkit_enabled", utils.is_buildkit_enabled), ("is_docker_rootless", utils.is_docker_rootless), ], ) @@ -458,6 +457,7 @@ def get_release(version: str) -> str: "14": "nutmeg", "15": "olive", "16": "palm", + "17": "quince", }[version.split(".", maxsplit=1)[0]] diff --git a/tutor/hooks/catalog.py b/tutor/hooks/catalog.py index 1b6b2fb9e6..56deed4e5e 100644 --- a/tutor/hooks/catalog.py +++ b/tutor/hooks/catalog.py @@ -251,8 +251,7 @@ def your_filter_callback(some_data): #: names must be prefixed with the plugin name in all-caps. CONFIG_UNIQUE: Filter[list[tuple[str, Any]], []] = Filter() - #: Use this filter to modify the ``docker build`` command. For instance, to replace - #: the ``build`` subcommand by ``buildx build``. + #: Use this filter to modify the ``docker build`` command. #: #: :parameter list[str] command: the full build command, including options and #: arguments. Note that these arguments do not include the leading ``docker`` command. @@ -335,7 +334,6 @@ def your_filter_callback(some_data): #: - ``HOST_USER_ID``: the numerical ID of the user on the host. #: - ``TUTOR_APP``: the app name ("tutor" by default), used to determine the dev/local project names. #: - ``TUTOR_VERSION``: the current version of Tutor. - #: - ``is_buildkit_enabled``: a boolean function that indicates whether BuildKit is available on the host. #: - ``iter_values_named``: a function to iterate on variables that start or end with a given string. #: - ``iter_mounts``: a function that yields compose-compatible bind-mounts for any given service. #: - ``iter_mounted_directories``: iterate on bind-mounted directory names. diff --git a/tutor/images.py b/tutor/images.py index 26f80b1824..ebab4ec505 100644 --- a/tutor/images.py +++ b/tutor/images.py @@ -4,8 +4,9 @@ def build(path: str, tag: str, *args: str) -> None: fmt.echo_info(f"Building image {tag}") build_command = ["build", f"--tag={tag}", *args, path] - if utils.is_buildkit_enabled(): - build_command.insert(0, "buildx") + # `buildx` can be removed once Tutor requires Docker v23+. At that point, BuildKit will be + # enabled by default for all Docker users. + build_command.insert(0, "buildx") command = hooks.Filters.DOCKER_BUILD_COMMAND.apply(build_command) utils.docker(*command) diff --git a/tutor/interactive.py b/tutor/interactive.py index 2e9e2f512f..820268a93a 100644 --- a/tutor/interactive.py +++ b/tutor/interactive.py @@ -18,7 +18,10 @@ def ask_questions(config: Config, run_for_prod: Optional[bool] = None) -> None: """ defaults = tutor_config.get_defaults() if run_for_prod is None: - run_for_prod = config.get("LMS_HOST") != "local.overhang.io" + run_for_prod = not config.get("LMS_HOST") in [ + "local.edly.io", + "local.overhang.io", + ] run_for_prod = click.confirm( fmt.question( "Are you configuring a production platform? " @@ -29,8 +32,8 @@ def ask_questions(config: Config, run_for_prod: Optional[bool] = None) -> None: ) if not run_for_prod: dev_values: Config = { - "LMS_HOST": "local.overhang.io", - "CMS_HOST": "studio.local.overhang.io", + "LMS_HOST": "local.edly.io", + "CMS_HOST": "studio.local.edly.io", "ENABLE_HTTPS": False, } fmt.echo_info( diff --git a/tutor/plugins/__init__.py b/tutor/plugins/__init__.py index 608421ade4..b2a3dcbf5a 100644 --- a/tutor/plugins/__init__.py +++ b/tutor/plugins/__init__.py @@ -17,13 +17,26 @@ @hooks.Actions.PLUGINS_LOADED.add() -def _convert_plugin_patches() -> None: +def _fill_patch_cache_on_load() -> None: + """ + This action is run after plugins have been loaded. + """ + _fill_patches_cache() + + +@hooks.Actions.PLUGIN_UNLOADED.add() +def _fill_patch_cache_on_unload(plugin: str, root: str, _config: Config) -> None: + """ + This action is run after plugins have been unloaded. + """ + _fill_patches_cache() + + +def _fill_patches_cache() -> None: """ Some patches are added as (name, content) tuples with the ENV_PATCHES filter. We convert these patches to add them to ENV_PATCHES_DICT. This makes it easier for end-user to declare patches, and it's more performant. - - This action is run after plugins have been loaded. """ ENV_PATCHES_DICT.clear() patches: t.Iterable[tuple[str, str]] = hooks.Filters.ENV_PATCHES.iterate() @@ -124,6 +137,6 @@ def unload(plugin: str) -> None: hooks.clear_all(context=hooks.Contexts.app(plugin).name) -@hooks.Actions.PLUGIN_UNLOADED.add(priority=50) +@hooks.Actions.PLUGIN_UNLOADED.add(priority=hooks.priorities.HIGH) def _unload_on_disable(plugin: str, _root: str, _config: Config) -> None: unload(plugin) diff --git a/tutor/plugins/openedx.py b/tutor/plugins/openedx.py index 0bf2c041bc..1ef4eff822 100644 --- a/tutor/plugins/openedx.py +++ b/tutor/plugins/openedx.py @@ -6,6 +6,16 @@ from tutor import bindmount from tutor import hooks +from tutor.__about__ import __version_suffix__ + + +@hooks.Filters.CONFIG_DEFAULTS.add() +def _set_openedx_common_version_in_nightly( + items: list[tuple[str, t.Any]] +) -> list[tuple[str, t.Any]]: + if __version_suffix__ == "nightly": + items.append(("OPENEDX_COMMON_VERSION", "master")) + return items @hooks.Filters.APP_PUBLIC_HOSTS.add() diff --git a/tutor/templates/apps/openedx/settings/partials/common_all.py b/tutor/templates/apps/openedx/settings/partials/common_all.py index b1d5a9f82a..c088b97b17 100644 --- a/tutor/templates/apps/openedx/settings/partials/common_all.py +++ b/tutor/templates/apps/openedx/settings/partials/common_all.py @@ -149,9 +149,11 @@ # These warnings are visible in simple commands and init tasks import warnings -from django.utils.deprecation import RemovedInDjango40Warning, RemovedInDjango41Warning -warnings.filterwarnings("ignore", category=RemovedInDjango40Warning) -warnings.filterwarnings("ignore", category=RemovedInDjango41Warning) + +from django.utils.deprecation import RemovedInDjango50Warning, RemovedInDjango51Warning +warnings.filterwarnings("ignore", category=RemovedInDjango50Warning) +warnings.filterwarnings("ignore", category=RemovedInDjango51Warning) + warnings.filterwarnings("ignore", category=DeprecationWarning, module="wiki.plugins.links.wiki_plugin") warnings.filterwarnings("ignore", category=DeprecationWarning, module="boto.plugin") warnings.filterwarnings("ignore", category=DeprecationWarning, module="botocore.vendored.requests.packages.urllib3._collections") @@ -189,6 +191,9 @@ "n": "{{ jwt_rsa_key.n|long_to_base64 }}", "p": "{{ jwt_rsa_key.p|long_to_base64 }}", "q": "{{ jwt_rsa_key.q|long_to_base64 }}", + "dq": "{{ jwt_rsa_key.dq|long_to_base64 }}", + "dp": "{{ jwt_rsa_key.dp|long_to_base64 }}", + "qi": "{{ jwt_rsa_key.invq|long_to_base64 }}", } ) JWT_AUTH["JWT_PUBLIC_SIGNING_JWK_SET"] = json.dumps( diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile index 351834cdd2..0aef8e81b8 100644 --- a/tutor/templates/build/openedx/Dockerfile +++ b/tutor/templates/build/openedx/Dockerfile @@ -1,11 +1,11 @@ -{% if is_buildkit_enabled() %}# syntax=docker/dockerfile:1.4{% endif %} +# syntax=docker/dockerfile:1 ###### Minimal image with base system requirements for most stages FROM docker.io/ubuntu:20.04 as minimal LABEL maintainer="Overhang.io " ENV DEBIAN_FRONTEND=noninteractive -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked{% endif %} \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt update && \ apt install -y build-essential curl git language-pack-en ENV LC_ALL en_US.UTF-8 @@ -14,8 +14,9 @@ ENV LC_ALL en_US.UTF-8 ###### Install python with pyenv in /opt/pyenv and create virtualenv in /openedx/venv FROM minimal as python # https://github.com/pyenv/pyenv/wiki/Common-build-problems#prerequisites -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git @@ -23,9 +24,9 @@ RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,shari # Install pyenv # https://www.python.org/downloads/ # https://github.com/pyenv/pyenv/releases -ARG PYTHON_VERSION=3.8.15 +ARG PYTHON_VERSION=3.8.18 ENV PYENV_ROOT /opt/pyenv -RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.17 --depth 1 +RUN git clone https://github.com/pyenv/pyenv $PYENV_ROOT --branch v2.3.29 --depth 1 # Install Python RUN $PYENV_ROOT/bin/pyenv install $PYTHON_VERSION @@ -68,13 +69,12 @@ FROM scratch as mnt-{{ name }} ###### Download extra locales to /openedx/locale/contrib/locale FROM minimal as locales -ARG OPENEDX_I18N_VERSION={{ OPENEDX_COMMON_VERSION }} -RUN cd /tmp \ - && curl -L -o openedx-i18n.tar.gz https://github.com/openedx/openedx-i18n/archive/$OPENEDX_I18N_VERSION.tar.gz \ - && tar xzf /tmp/openedx-i18n.tar.gz \ - && mkdir -p /openedx/locale/contrib \ - && mv openedx-i18n-*/edx-platform/locale /openedx/locale/contrib \ - && rm -rf openedx-i18n* +ARG OPENEDX_I18N_REPOSITORY=https://github.com/openedx/openedx-i18n.git +ARG OPENEDX_I18N_VERSION=release/quince +ADD --keep-git-dir=true $OPENEDX_I18N_REPOSITORY#$OPENEDX_I18N_VERSION /tmp/openedx-i18n +RUN mkdir --parents /openedx/locale && \ + mv /tmp/openedx-i18n/edx-platform/locale /openedx/locale/contrib && \ + rm -rf /tmp/openedx-i18n/ ###### Install python requirements in virtualenv FROM python as python-requirements @@ -82,43 +82,40 @@ ENV PATH /openedx/venv/bin:${PATH} ENV VIRTUAL_ENV /openedx/venv/ ENV XDG_CACHE_HOME /openedx/.cache -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update \ && apt install -y software-properties-common libmysqlclient-dev libxmlsec1-dev libgeos-dev # Install the right version of pip/setuptools -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ # https://pypi.org/project/setuptools/ # https://pypi.org/project/pip/ # https://pypi.org/project/wheel/ - setuptools==67.6.1 pip==23.0.1. wheel==0.40.0 - -# Install missing py2neo package that was abruptly trimmed from pypi -RUN pip install https://github.com/overhangio/py2neo/releases/download/2021.2.3/py2neo-2021.2.3.tar.gz + setuptools==68.2.2 pip==23.2.1. wheel==0.41.2 # Install base requirements -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /requirements/edx/base.txt /openedx/edx-platform/requirements/edx/base.txt -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ - --mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r /openedx/edx-platform/requirements/edx/base.txt +RUN --mount=type=bind,from=edx-platform,source=/requirements/edx/base.txt,target=/openedx/edx-platform/requirements/edx/base.txt \ + --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r /openedx/edx-platform/requirements/edx/base.txt # Install extra requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install \ +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install \ # Use redis as a django cache https://pypi.org/project/django-redis/ - django-redis==5.2.0 \ + django-redis==5.4.0 \ # uwsgi server https://pypi.org/project/uWSGI/ - uwsgi==2.0.21 + uwsgi==2.0.22 {{ patch("openedx-dockerfile-post-python-requirements") }} -# Install private requirements: this is useful for installing custom xblocks. -COPY ./requirements/ /openedx/requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}cd /openedx/requirements/ \ - && touch ./private.txt \ - && pip install -r ./private.txt +# Install scorm xblock +RUN pip install "openedx-scorm-xblock>=17.0.0,<18.0.0" -{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install '{{ extra_requirements }}' +{% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %} +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install '{{ extra_requirements }}' {% endfor %} ###### Install nodejs with nodeenv in /openedx/nodeenv @@ -134,20 +131,19 @@ RUN nodeenv /openedx/nodeenv --node=16.14.0 --prebuilt # Install nodejs requirements ARG NPM_REGISTRY={{ NPM_REGISTRY }} WORKDIR /openedx/edx-platform -{% if not is_buildkit_enabled() %} -COPY --from=edx-platform /package.json /openedx/edx-platform/package.json -COPY --from=edx-platform /package-lock.json /openedx/edx-platform/package-lock.json -{% endif %} -RUN {% if is_buildkit_enabled() %}--mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ +RUN --mount=type=bind,from=edx-platform,source=/package.json,target=/openedx/edx-platform/package.json \ --mount=type=bind,from=edx-platform,source=/package-lock.json,target=/openedx/edx-platform/package-lock.json \ - --mount=type=cache,target=/root/.npm,sharing=shared {% endif %}npm clean-install --no-audit --registry=$NPM_REGISTRY + --mount=type=bind,from=edx-platform,source=/scripts/copy-node-modules.sh,target=/openedx/edx-platform/scripts/copy-node-modules.sh \ + --mount=type=cache,target=/root/.npm,sharing=shared \ + npm clean-install --no-audit --registry=$NPM_REGISTRY ###### Production image with system and python requirements FROM minimal as production # Install system requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update \ && apt install -y gettext gfortran graphviz graphviz-dev libffi-dev libfreetype6-dev libgeos-dev libjpeg8-dev liblapack-dev libmysqlclient-dev libpng-dev libsqlite3-dev libxmlsec1-dev lynx mysql-client ntp pkg-config rdfind # From then on, run as unprivileged "app" user @@ -158,12 +154,11 @@ RUN useradd --no-log-init --home-dir /openedx --create-home --shell /bin/bash -- USER ${APP_USER_ID} # https://hub.docker.com/r/powerman/dockerize/tags -COPY {% if is_buildkit_enabled() %}--link {% endif %}--from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize +COPY --link --from=docker.io/powerman/dockerize:0.19.0 /usr/local/bin/dockerize /usr/local/bin/dockerize COPY --chown=app:app --from=edx-platform / /openedx/edx-platform COPY --chown=app:app --from=locales /openedx/locale /openedx/locale COPY --chown=app:app --from=python /opt/pyenv /opt/pyenv COPY --chown=app:app --from=python-requirements /openedx/venv /openedx/venv -COPY --chown=app:app --from=python-requirements /openedx/requirements /openedx/requirements COPY --chown=app:app --from=python-requirements /mnt /mnt COPY --chown=app:app --from=nodejs-requirements /openedx/nodeenv /openedx/nodeenv COPY --chown=app:app --from=nodejs-requirements /openedx/edx-platform/node_modules /openedx/node_modules @@ -259,16 +254,19 @@ FROM production as development # Install useful system requirements (as root) USER root -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked {% endif %}apt update && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + apt update && \ apt install -y vim iputils-ping dnsutils telnet USER app # Install dev python requirements -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install -r requirements/edx/development.txt +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install -r requirements/edx/development.txt # https://pypi.org/project/ipdb/ -# https://pypi.org/project/ipython -RUN {% if is_buildkit_enabled() %}--mount=type=cache,target=/openedx/.cache/pip,sharing=shared {% endif %}pip install ipdb==0.13.13 ipython==8.12.0 +# https://pypi.org/project/ipython (8.12.x for Python 3.8) +RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \ + pip install ipdb==0.13.13 ipython==8.12.3 {# Re-install mounted requirements, otherwise they will be superseded by upstream reqs #} {% for name in iter_mounted_directories(MOUNTS, "openedx") %} diff --git a/tutor/templates/build/openedx/requirements/private-sample.txt b/tutor/templates/build/openedx/requirements/private-sample.txt deleted file mode 100644 index a0ea63a612..0000000000 --- a/tutor/templates/build/openedx/requirements/private-sample.txt +++ /dev/null @@ -1,6 +0,0 @@ -# Add your additional requirements, such as xblocks, to this file. For -# requirements coming from private repositories, clone the repository to this -# folder and then add your requirement with the `-e` flag. Ex: -# -# git clone git@myserver:myprivaterepo.git -# echo "-e ./myprivaterepo/" >> private.txt diff --git a/tutor/templates/build/openedx/revisions.yml b/tutor/templates/build/openedx/revisions.yml index 34724a9707..c01e271e53 100644 --- a/tutor/templates/build/openedx/revisions.yml +++ b/tutor/templates/build/openedx/revisions.yml @@ -1 +1 @@ -EDX_PLATFORM_REVISION: palm +EDX_PLATFORM_REVISION: quince diff --git a/tutor/templates/config/base.yml b/tutor/templates/config/base.yml index e34095a61b..b1d6a14afa 100644 --- a/tutor/templates/config/base.yml +++ b/tutor/templates/config/base.yml @@ -6,8 +6,9 @@ MYSQL_ROOT_PASSWORD: "{{ 8|random_string }}" OPENEDX_MYSQL_PASSWORD: "{{ 8|random_string }}" OPENEDX_SECRET_KEY: "{{ 24|random_string }}" PLUGINS: - # The MFE plugin is required + # enabled by default - mfe + - indigo PLUGIN_INDEXES: # Indexes in this list will be suffixed with the Open edX named version and # "plugins.yml". E.g: https://overhang.io/tutor/main/olive/plugins.yml diff --git a/tutor/templates/config/defaults.yml b/tutor/templates/config/defaults.yml index c75332812a..9ab1967ada 100644 --- a/tutor/templates/config/defaults.yml +++ b/tutor/templates/config/defaults.yml @@ -12,12 +12,18 @@ DOCKER_COMPOSE_VERSION: "3.7" DOCKER_REGISTRY: "docker.io/" DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION }}" DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev:{{ TUTOR_VERSION }}" -DOCKER_IMAGE_CADDY: "docker.io/caddy:2.6.4" -DOCKER_IMAGE_ELASTICSEARCH: "docker.io/elasticsearch:7.17.9" -DOCKER_IMAGE_MONGODB: "docker.io/mongo:4.4.22" +# https://hub.docker.com/_/caddy/tags +DOCKER_IMAGE_CADDY: "docker.io/caddy:2.7.4" +# https://hub.docker.com/_/elasticsearch/tags +DOCKER_IMAGE_ELASTICSEARCH: "docker.io/elasticsearch:7.17.13" +# https://hub.docker.com/_/mongo/tags +DOCKER_IMAGE_MONGODB: "docker.io/mongo:4.4.25" +# https://hub.docker.com/_/mysql/tags DOCKER_IMAGE_MYSQL: "docker.io/mysql:8.1.0" DOCKER_IMAGE_PERMISSIONS: "{{ DOCKER_REGISTRY }}overhangio/openedx-permissions:{{ TUTOR_VERSION }}" -DOCKER_IMAGE_REDIS: "docker.io/redis:7.0.11" +# https://hub.docker.com/_/redis/tags +DOCKER_IMAGE_REDIS: "docker.io/redis:7.2.1" +# https://hub.docker.com/r/devture/exim-relay/tags DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.96-r1-0" EDX_PLATFORM_REPOSITORY: "https://github.com/openedx/edx-platform.git" EDX_PLATFORM_VERSION: "{{ OPENEDX_COMMON_VERSION }}" @@ -53,9 +59,8 @@ OPENEDX_LMS_UWSGI_WORKERS: 2 OPENEDX_MYSQL_DATABASE: "openedx" OPENEDX_MYSQL_USERNAME: "openedx" # the common version will be automatically set to "master" in the nightly branch -OPENEDX_COMMON_VERSION: "open-release/palm.4" -OPENEDX_EXTRA_PIP_REQUIREMENTS: - - "openedx-scorm-xblock>=16.0.0,<17.0.0" +OPENEDX_COMMON_VERSION: "open-release/quince.1" +OPENEDX_EXTRA_PIP_REQUIREMENTS: [] MYSQL_HOST: "mysql" MYSQL_PORT: 3306 MYSQL_ROOT_USERNAME: "root" diff --git a/tutor/templates/dev/docker-compose.jobs.yml b/tutor/templates/dev/docker-compose.jobs.yml index c9df4e2a12..5e82a455aa 100644 --- a/tutor/templates/dev/docker-compose.jobs.yml +++ b/tutor/templates/dev/docker-compose.jobs.yml @@ -16,8 +16,6 @@ x-openedx-job-service: - ../apps/openedx/config:/openedx/config:ro # theme files - ../build/openedx/themes:/openedx/themes - # editable requirements - - ../build/openedx/requirements:/openedx/requirements services: diff --git a/tutor/templates/dev/docker-compose.yml b/tutor/templates/dev/docker-compose.yml index 7c864358fc..90fbe842bd 100644 --- a/tutor/templates/dev/docker-compose.yml +++ b/tutor/templates/dev/docker-compose.yml @@ -12,8 +12,6 @@ x-openedx-service: - ../apps/openedx/config:/openedx/config:ro # theme files - ../build/openedx/themes:/openedx/themes - # editable requirements - - ../build/openedx/requirements:/openedx/requirements services: permissions: diff --git a/tutor/templates/jobs/init/cms.sh b/tutor/templates/jobs/init/cms.sh index 2f732da22f..1d420a48ab 100644 --- a/tutor/templates/jobs/init/cms.sh +++ b/tutor/templates/jobs/init/cms.sh @@ -11,4 +11,8 @@ if [ -d /openedx/data/uploads/ ]; then mv /openedx/data/uploads/* /openedx/media/ rm -rf /openedx/data/uploads/ fi -fi \ No newline at end of file +fi + +# Create waffle switches to enable some features, if they have not been explicitly defined before +# Copy-paste of units in Studio (highly requested new feature, but defaults to off in Quince) +(./manage.py cms waffle_flag --list | grep contentstore.enable_copy_paste_units) || ./manage.py lms waffle_flag --create contentstore.enable_copy_paste_units --everyone diff --git a/tutor/templates/local/docker-compose.prod.yml b/tutor/templates/local/docker-compose.prod.yml index 552fa44dd0..f782a9e68a 100644 --- a/tutor/templates/local/docker-compose.prod.yml +++ b/tutor/templates/local/docker-compose.prod.yml @@ -20,7 +20,7 @@ services: networks: default: # These aliases are for internal communication between containers when running locally - # with *.local.overhang.io hostnames. + # with *.local.edly.io hostnames. aliases: - "{{ LMS_HOST }}" {{ patch("local-docker-compose-caddy-aliases")|indent(10) }} diff --git a/tutor/utils.py b/tutor/utils.py index e81823f12a..cdb082f38a 100644 --- a/tutor/utils.py +++ b/tutor/utils.py @@ -173,25 +173,6 @@ def docker(*command: str) -> int: return execute("docker", *command) -@lru_cache(maxsize=None) -def is_buildkit_enabled() -> bool: - """ - A helper function to determine whether we can run `docker buildx` with BuildKit. - """ - # First, we respect the DOCKER_BUILDKIT environment variable - enabled_by_env = { - "1": True, - "0": False, - }.get(os.environ.get("DOCKER_BUILDKIT", "")) - if enabled_by_env is not None: - return enabled_by_env - try: - subprocess.run(["docker", "buildx", "version"], capture_output=True, check=True) - return True - except subprocess.CalledProcessError: - return False - - @lru_cache(maxsize=None) def is_docker_rootless() -> bool: """