[Python][CI] test_dlpack is failing on some nightly jobs allocating unexpected memory

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Several jobs:

• test-conda-python-3.11-pandas-latest-numpy-latest
• test-conda-python-3.11-pandas-nightly-numpy-nightly
• test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
• test-conda-python-3.9

are failing for the test_dlpack test with some variants of:

 ________________________ test_dlpack[value_type6-int32] ________________________

args = (), kwargs = {'np_type_str': 'int32', 'value_type': DataType(int32)}
allocated_bytes = 7440
@py_assert1 = <cyfunction total_allocated_bytes at 0x7fc798db2b70>
@py_assert3 = 3728, @py_assert5 = False
@py_format7 = '3728\n{3728 = <cyfunction total_allocated_bytes at 0x7fc798db2b70>\n{<cyfunction total_allocated_bytes at 0x7fc798db2b70> = pa.total_allocated_bytes\n}()\n} == 7440'
@py_format9 = 'assert 3728\n{3728 = <cyfunction total_allocated_bytes at 0x7fc798db2b70>\n{<cyfunction total_allocated_bytes at 0x7fc798db2b70> = pa.total_allocated_bytes\n}()\n} == 7440'

    @wraps(f)
    def wrapper(*args, **kwargs):
        allocated_bytes = pa.total_allocated_bytes()
        try:
            return f(*args, **kwargs)
        finally:
>           assert pa.total_allocated_bytes() == allocated_bytes
E           assert 3728 == 7440
E            +  where 3728 = <cyfunction total_allocated_bytes at 0x7fc798db2b70>()
E            +    where <cyfunction total_allocated_bytes at 0x7fc798db2b70> = pa.total_allocated_bytes

Component(s)

Continuous Integration, Python

[jorisvandenbossche]

This started to fail on the main CI as well (eg #44720).

I don't directly see any difference in the conda env between the failing build and the last build on main (which is still green) ..

[raulcd]

I can't reproduce locally with:

$ PYTHON=3.11 PANDAS=latest NUMPY=latest docker compose build conda-python-pandas
$ PYTHON=3.11 PANDAS=latest NUMPY=latest docker compose run conda-python-pandas

The test is successful:

opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/tests/test_dlpack.py ............s
...
7539 passed, 231 skipped, 23 xfailed, 2 xpassed, 15 warnings in 202.04s (0:03:22) 

The only skipped test is the cuda requiring one, the same skipped in CI:

SKIPPED [1] opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/tests/test_dlpack.py:133: could not import 'pyarrow.cuda': No module named 'pyarrow._cuda'

[raulcd]

Issue resolved by pull request 44793
#44793