Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bidscoin pip install fails on apple silicon #189

Closed
dkp opened this issue Jun 30, 2023 · 25 comments
Closed

bidscoin pip install fails on apple silicon #189

dkp opened this issue Jun 30, 2023 · 25 comments

Comments

@dkp
Copy link

dkp commented Jun 30, 2023

Describe the bug
bidscoin environment creation fails on Apple Silicon (though it worked on my intel mac).

The install hangs here:

Collecting PyQt5>=5.12.1 (from bidscoin)
  Using cached PyQt5-5.15.9.tar.gz (3.2 MB)
  Installing build dependencies ... done

StackOverflow reports that pyqt5 is not compatible with the ARM architecture, but PyQt6 IS compatible:
https://stackoverflow.com/questions/71046800/how-to-install-pyqt5-on-m1-arm64-architecture

To reproduce
On Apple Silicon:

conda create -n bidscoin python=3.10
conda activate bidscoin
pip install bidscoin[all] 

Expected behavior
I hoped it would install ; (

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Jun 30, 2023 via email

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Jun 30, 2023 via email

@dkp
Copy link
Author

dkp commented Jun 30, 2023

Wow. I think it may just have worked!

python setup.py install
...
Finished processing dependencies for bidscoin==4.0.0

pyqt6_issues.txt

bidscoin gives me a nice usage message, however, when I try to start the GUI, I run into a problem with PyQT ; (
I have attached the messages output by bidscoin -t
However, I also tried to run the gui and it crashed with complaints about the same library.

@marcelzwiers
Copy link
Collaborator

Could it be that you have both PyQt5 and PyQt6 installed? Or did you install the qt6 branch in it's own new environment?

@marcelzwiers
Copy link
Collaborator

I think this library error is a bit out of my scope, @bendhouseart have you perhaps seen this?

@dkp
Copy link
Author

dkp commented Jun 30, 2023

My Python knowledge is wobbly. Maybe you can help me understand where to go from here. Honestly, i can't say in PyQt5 is lurking somewhere.

I cloned the qt6 branch:

git clone --branch qt6 https://github.com/Donders-Institute/bidscoin.git

I tried creating a venv and building there with python3 setup.py install
Although it claims to finish, it is clearly has problems:

(venv_qt6) [lolo:bidscoin_qt6:qt6 586] $ bidscoin -t
Traceback (most recent call last):
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/bin/bidscoin", line 33, in <module>
    sys.exit(load_entry_point('bidscoin==4.0.0', 'console_scripts', 'bidscoin')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/bin/bidscoin", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/lib/python3.11/site-packages/bidscoin-4.0.0-py3.11.egg/bidscoin/bcoin.py", line 33, in <module>
    from ruamel.yaml import YAML
ImportError: cannot import name 'YAML' from 'ruamel.yaml' (/Users/dpat/projects/Testing/neuroimaging/dicom_conversion/bidscoin/code/venv_qt6/lib/python3.11/site-packages/ruamel.yaml.clib-0.2.7-py3.11-macosx-13-arm64.egg/ruamel/yaml/__init__.py)

Outside a virtual env has the same problems.
This is installing with brew's python, so I uninstalled:

Found existing installation: bidscoin 4.0.0
Uninstalling bidscoin-4.0.0:
  Would remove:
    /opt/homebrew/bin/bidscoin
    /opt/homebrew/bin/bidscoiner
    /opt/homebrew/bin/bidseditor
    /opt/homebrew/bin/bidsmapper
    /opt/homebrew/bin/bidsparticipants
    /opt/homebrew/bin/deface
    /opt/homebrew/bin/dicomsort
    /opt/homebrew/bin/echocombine
    /opt/homebrew/bin/medeface
    /opt/homebrew/bin/physio2tsv
    /opt/homebrew/bin/plotphysio
    /opt/homebrew/bin/rawmapper
    /opt/homebrew/bin/skullstrip
    /opt/homebrew/bin/slicereport
    /opt/homebrew/lib/python3.11/site-packages/bidscoin-4.0.0-py3.11.egg
Proceed (Y/n)? Y
  Successfully uninstalled bidscoin-4.0.0

@marcelzwiers
Copy link
Collaborator

I think installing bidscoin-qt6 in a virtual environment, as you did, should just work. I don't own a mac, so I can't replicate your issue myself, but I'll try to find a colleague with a mac who can test this out for me. In the meantime, perhaps this is of help?
https://forum.qt.io/topic/136786/how-to-fix-the-error-qtwidgets-abi3-so-reason-image-not-found-in-python-on-mac-os-x-catalina/2

@marcelzwiers
Copy link
Collaborator

@dkp
Copy link
Author

dkp commented Jul 3, 2023

Thank you for the suggestions. I tried the regular bidscoin distribution on linux (intel) the other day and got errors there too (though completely different):

x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/dpat/projects/Testing/bidscoin/venv_bc/include -I/usr/include/python3.11 -c traits/ctraits.c -o build/temp.linux-x86_64-cpython-311/traits/ctraits.o
      error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for traits
Failed to build traits
ERROR: Could not build wheels for traits, which is required to install pyproject.toml-based projects

I'll probably try apptainer on our HPC next, but it may be a little while before I get to it. I'll let you know how it goes. I appreciate your willingness to help.

@marcelzwiers
Copy link
Collaborator

I have never seen this message and have no idea what could be the cause of it. I'll look into it tomorrow...

@marcelzwiers
Copy link
Collaborator

That traits error seems rather fundamental, can you install other packages without error? E.g. pip install pydicom?

@dkp
Copy link
Author

dkp commented Jul 6, 2023

Aha, thank you for your patience. Problem solved on Linux. There were two factors, I think:

  • For conda, it is important that it be installed for the user (we had a system with conda installed "globally" and this then resulted in permission problems.
  • Again, for conda, it is critical to explicitly specify the version of python when creating the environment:
    e.g., conda create -n bidscoin python=3.10

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Jul 6, 2023

  • For conda, it is important that it be installed for the user (we had a system with conda installed "globally" and this then resulted in permission problems.

You mean that you have to install it in a (e.g. conda) virtual environment? Sure, that is not just true for BIDScoin, but for any python library you install

  • Again, for conda, it is critical to explicitly specify the version of python when creating the environment:
    e.g., conda create -n bidscoin python=3.10

Although I have seen this clunky conda behaviour too in the past, it is certainly not true in general (in other OS / anaconda versions it works just fine)

@marcelzwiers
Copy link
Collaborator

Which Linux version is that? Some old CentOS version by any chance?

@dkp
Copy link
Author

dkp commented Jul 6, 2023

This is ubuntu 22.0.2:
I may misunderstand what I'm encountering, but I'm talking about who installs CONDA not who uses conda to create an environment. In the case of our local linux box, my husband set everything up. He installed conda in /usr/local/src. But it seemed that carried all sorts of permissions baggage with it such that my account had some, but not all, relevant permissions for setting up individual conda environments. Anyhow, part of the "fix" was to install miniconda for each user separately.

I've always set up conda for myself, so I was not aware of what issues might ensue from a global install.

@marcelzwiers
Copy link
Collaborator

Thanks, I didn't know that. Glad you figured that out :)

@dkp
Copy link
Author

dkp commented Jul 6, 2023

Well, maybe there is some way around the problem, I just don't know what that is yet.

@bendhouseart
Copy link
Contributor

Just got back from summer break, so apologies for the late reply.

I think this library error is a bit out of my scope, @bendhouseart have you perhaps seen this?

Hi @marcelzwiers don't recall running into the specific errors in the attached text file. But, happy to test this out on Apple Silicon for you and @dkp

Currently getting hanging doing the install w/ dependencies for pyqt5 via python 3.10:

Collecting bidscoin
  Using cached bidscoin-4.0.0-py3-none-any.whl (655 kB)
Collecting pandas (from bidscoin)
  Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)
Collecting matplotlib (from bidscoin)
  Downloading matplotlib-3.7.2-cp310-cp310-macosx_11_0_arm64.whl (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 8.9 MB/s eta 0:00:00
Collecting numpy (from bidscoin)
  Downloading numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 17.7 MB/s eta 0:00:00
Collecting pydicom>=2 (from bidscoin)
  Downloading pydicom-2.4.1-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 20.5 MB/s eta 0:00:00
Collecting PyQt5>=5.12.1 (from bidscoin)
  Downloading PyQt5-5.15.9.tar.gz (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 21.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... -

That said, installing from here git clone --branch qt6 https://github.com/Donders-Institute/bidscoin.git, does reproduce the ruamel.yaml error.... so that's something.

Really frustrating when even the example in their docs fails miserably. But, that's to be expected I think when the docs are a year out of date at least:

>>> from ruamel.yaml import YAML
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'YAML' from 'ruamel.yaml' (/Users/galassiae/Projects/TestInstallBidscoinFromPIPversionQT6/bidscoin/venv/lib/python3.10/site-packages/ruamel.yaml.clib-0.2.7-py3.10-macosx-12-arm64.egg/ruamel/yaml/__init__.py)

If I roll back to rumel.yaml version 0.17.0 I no longer get that error:

(venv) galassiae@MH02276145MLI bidscoin % pip install ruamel.yaml==0.17.0
Collecting ruamel.yaml==0.17.0
  Downloading ruamel.yaml-0.17.0-py2.py3-none-any.whl (101 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.5/101.5 kB 1.9 MB/s eta 0:00:00
WARNING: The candidate selected for download or install is a yanked version: 'ruamel-yaml' candidate (version 0.17.0 at https://files.pythonhosted.org/packages/69/b8/ea30b2f7a38680be7c823428d502e009e2f2a9d7122e57221bf6397bc77f/ruamel.yaml-0.17.0-py2.py3-none-any.whl (from https://pypi.org/simple/ruamel-yaml/))
Reason for being yanked: would still install for Python 2
Installing collected packages: ruamel.yaml
Successfully installed ruamel.yaml-0.17.0

[notice] A new release of pip is available: 23.1.2 -> 23.2
[notice] To update, run: pip install --upgrade pip
(venv) galassiae@MH02276145MLI bidscoin % python
Python 3.10.11 (main, Apr  7 2023, 07:24:47) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ruamel.yaml import YAML
>>> 

It might be worth rolling back to an earlier version of this library or ditching it for pyYAML or something else.

@marcelzwiers I'm going to fiddle around with the qt6 branch mentioned earlier, let me know if you would rather update it from master/current release before I get too carried away.

@bendhouseart
Copy link
Contributor

Alright, moving over discussion from mattermost to here so it's more visible.

Turns out I wasn't just "tired and out of it" pip install . has some real issues with python 3.11 and Apple. Seems to time out after about an hour of failing to resolve a new pyproject.toml/lock file, see:

  Using cached PyQt5-5.15.9.tar.gz (3.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... -

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Aug 23, 2023

@bendhouseart Well, I also have trouble running tox with python 3.11, but that's because of pypet2bids depending on a specific python version, i.e. from pypet2bids/pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.8,<=3.11.1"

It would make my life easier if you would relax it a bit and take out the point release (I think that's a harmless thing to do): python = ">=3.8,<=3.11"

Here's the error:

py311: install_package_deps> python -I -m pip install 'pypet2bids>=1.0.12'
ERROR: Ignored the following versions that require a different python version: 0.0.10 Requires-Python >3.7.1,<3.10; 0.0.11 Requires-Python >3.7.1,<3.10; 0.0.12 Requires-Python >3.7.1,<3.10; 0.0.13 Requires-Python >3.7.1,<3.10; 0.0.14 Requires-Python >3.7.1,<3.10; 0.0.15 Requires-Python >3.7.1,<3.10; 0.0.4 Requires-Python >3.7.1,<3.10; 0.0.6 Requires-Python >3.7.1,<3.10; 0.0.7 Requires-Python >3.7.1,<3.10; 0.0.8 Requires-Python >3.7.1,<3.10; 0.0.9 Requires-Python >3.7.1,<3.10; 1.0.0 Requires-Python >3.7.1,<3.10; 1.0.10 Requires-Python >=3.8,<3.11; 1.0.12 Requires-Python >=3.8,<=3.11.1; 1.0.2 Requires-Python >3.7.1,<3.10; 1.0.5 Requires-Python >=3.8,<3.11; 1.0.7 Requires-Python >=3.8,<3.11; 1.0.8 Requires-Python >=3.8,<3.11; 1.0.9 Requires-Python >=3.8,<3.11; 1.1.0 Requires-Python >=3.8,<=3.11.1; 1.1.1 Requires-Python >=3.8,<=3.11.1; 1.1.2 Requires-Python >=3.8,<=3.11.1; 1.2.2 Requires-Python >=3.8,<=3.11.1; 1.2.3 Requires-Python >=3.8,<=3.11.1
ERROR: Could not find a version that satisfies the requirement pypet2bids>=1.0.12 (from versions: 1.0.4)
ERROR: No matching distribution found for pypet2bids>=1.0.12

@marcelzwiers
Copy link
Collaborator

@bendhouseart I chose ruamel.yaml over pyyaml because pyyaml still doesn't support the 10 year old release of yaml v1.2. Generally, it has been very good to me, including supporting writing back the comments (which I think pyyaml just drops). Which version of ruamel.yaml wasn't working for you? I suspect it is the compiled part (see ruamel.yaml.clib-0.2.7-py3.10-macosx-12-arm64.egg) that is giving problems... Perhaps you can use conda-forge?

https://anaconda.org/conda-forge/ruamel.yaml

@marcelzwiers
Copy link
Collaborator

@bendhouseart Perhaps you can file a ruamel.yaml ticket? I think the author is very responsive to good tickets
https://sourceforge.net/p/ruamel-yaml/tickets/

@marcelzwiers
Copy link
Collaborator

@bendhouseart I've been reading up a bit on how to use python on MacOS and learned that you probably don't want to use brew to create virtual environments, but use conda, venv or pyenv or so. I saw in the qt6 log that the yaml library was reading the yaml file just fine. Also, using setup.py is not always the same as using pip, so to be safe, I'd use the latter to make sure all dependencies are installed

@marcelzwiers
Copy link
Collaborator

marcelzwiers commented Aug 28, 2023

Ok, I managed to borrow a brand new M2 MacBook pro (out of the box). I did the following

  1. Installed a standard python 3.11.5 interpreter using the universal installer from here: https://www.python.org/downloads/macos/
  2. I then installed SSL certificates for python by running the Install Certificates.command in the python application folder (this is not really needed but used for downloading tutorial data and checking the latest pypi version)
  3. I created a standard virtual environment using: python3 -m venv bidscoin
  4. I installed the qt6 branch using: source bidscoin/bin/activate; python3 -m pip install git+https://github.com/Donders-Institute/bidscoin@qt6
  5. Running the test bidscoin -t just works normally (except for plugin errors, which I didn't install)
  6. Runing the bidsmapper and bidseditor also work normally (with GUI)

So no ruamel.yaml error and no pyqt6 error. It seems to me the problems are caused by homebrew

@marcelzwiers
Copy link
Collaborator

I'm closing this issue because it is due to the user installation. Feel free to re-open it if needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants