ResolutionTooDeep error in during installation of the latest xgboost-ray from PyPI in Python 3.9 containers

[mirekphd]

For the past couple of weeks we are having problems installing xgboost_ray from PyPi in all of our CPU-only and GPU-enabled containers with Python 3.9.

Two very similar containers differing only in one respect - having newer Python version (3.11 vs. 3.9), but sharing the same pip version (23.3.1) and the same list of Python packages, including XGBoost pinned to the latest version less than 2.0.0 (i.e. to 1.7.6) (the packages list is re-used via a Jenkins shared library) work fine (pip installs ray 2.8.0 and xgboost-ray 0.1.19 (currently latest).

In Python 3.9 containers (with either Jupyter or VS Code Server) the installation of xgboost_ray hangs after approximately 10 hours (during which progressively older versions of the package are being downloaded as newer versions fail to be reconciled with other packages), finally reaching version 0.0.2 and failing with the pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000 error.

The full stack trace (from Jenkins logs):

21:56:57  Collecting xgboost-ray (from -r /tmp/python-packages/pypi-packages.txt (line 184))
21:56:57    Downloading xgboost_ray-0.1.18-py3-none-any.whl.metadata (492 bytes)
22:41:18    Downloading xgboost_ray-0.1.17-py3-none-any.whl.metadata (492 bytes)
23:25:39    Downloading xgboost_ray-0.1.16-py3-none-any.whl.metadata (492 bytes)
00:10:46    Downloading xgboost_ray-0.1.15-py3-none-any.whl (138 kB)
00:10:46       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.1/138.1 kB 18.4 MB/s eta 0:00:00
00:56:37    Downloading xgboost_ray-0.1.14-py3-none-any.whl (138 kB)
00:56:37       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.1/138.1 kB 20.3 MB/s eta 0:00:00
01:42:43    Downloading xgboost_ray-0.1.13-py3-none-any.whl (137 kB)
01:42:43       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.7/137.7 kB 13.3 MB/s eta 0:00:00
02:27:50    Downloading xgboost_ray-0.1.12-py3-none-any.whl (137 kB)
02:27:50       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.3/137.3 kB 16.4 MB/s eta 0:00:00
03:11:41    Downloading xgboost_ray-0.1.11-py3-none-any.whl (137 kB)
03:11:41       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.1/137.1 kB 15.3 MB/s eta 0:00:00
03:56:47    Downloading xgboost_ray-0.1.10-py3-none-any.whl (137 kB)
03:56:47       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.5/137.5 kB 9.1 MB/s eta 0:00:00
04:41:53    Downloading xgboost_ray-0.1.9-py3-none-any.whl (230 kB)
04:41:53       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 230.6/230.6 kB 17.7 MB/s eta 0:00:00
05:26:15    Downloading xgboost_ray-0.1.8-py3-none-any.whl (138 kB)
05:26:15       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.4/138.4 kB 16.0 MB/s eta 0:00:00
06:11:51    Downloading xgboost_ray-0.1.7-py3-none-any.whl (138 kB)
06:11:51       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.3/138.3 kB 17.7 MB/s eta 0:00:00
06:56:27    Downloading xgboost_ray-0.1.6-py3-none-any.whl (137 kB)
06:56:27       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.2/137.2 kB 20.2 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.1.5-py3-none-any.whl (136 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.4/136.4 kB 14.3 MB/s eta 0:00:00
07:42:18  INFO: pip is looking at multiple versions of xgboost-ray to determine which version is compatible with other requirements. This could take a while.
07:42:18    Downloading xgboost_ray-0.1.4-py3-none-any.whl (135 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 kB 53.5 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.1.3-py3-none-any.whl (129 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.9/129.9 kB 165.6 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.1.2-py3-none-any.whl (121 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.5/121.5 kB 104.8 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.1.1-py3-none-any.whl (119 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.5/119.5 kB 121.4 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.1.0-py3-none-any.whl (76 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.8/76.8 kB 313.5 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.0.5-py3-none-any.whl (72 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.8/72.8 kB 314.9 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.0.4-py3-none-any.whl (63 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.5/63.5 kB 313.8 MB/s eta 0:00:00
07:42:18  INFO: pip is still looking at multiple versions of xgboost-ray to determine which version is compatible with other requirements. This could take a while.
07:42:18    Downloading xgboost_ray-0.0.3-py3-none-any.whl (49 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.9/49.9 kB 263.0 MB/s eta 0:00:00
07:42:18    Downloading xgboost_ray-0.0.2-py3-none-any.whl (47 kB)
07:42:18       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.2/47.2 kB 292.4 MB/s eta 0:00:00
07:45:55  �[91mERROR: Exception:
07:45:55  Traceback (most recent call last):
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
07:45:55      status = run_func(*args)
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
07:45:55      return func(self, options, args)
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
07:45:55      requirement_set = resolver.resolve(
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
07:45:55      result = self._result = resolver.resolve(
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
07:45:55      state = resolution.resolve(requirements, max_rounds=max_rounds)
07:45:55    File "/opt/conda/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
07:45:55      raise ResolutionTooDeep(max_rounds)
07:45:55  pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000
07:45:55  �[0mThe command '/bin/bash -o pipefail -c pip install -r /tmp/python-packages/pypi-packages.txt --no-cache-dir' returned a non-zero code: 2