Install grumpy-tools before building grumpy-runtime #140

abitrolly · 2019-10-03T14:03:18Z

pip2 install grumpy-runtime fails with the ERROR
#113 (comment)
because it relies on grumpy-tools.

grumpy-tools packages is specified in setup.py as a
dependency, and it doesn't work, because for the build process to
succeed, the dependency should already exist at the time setup.py
is executed.

The official workaround is pyproject.toml from PEP-518
pypa/pip#2381

abitrolly · 2019-10-03T17:35:07Z

This still doesn't fix the error. pip claims that it installs build dependencies.

$ pipenv clean
$ pipenv run python setup.py sdist
...
copying pyproject.toml -> grumpy-runtime-0.3.0
...
$  pipenv run pip install dist/grumpy-runtime-0.3.0.tar.gz 
Processing ./dist/grumpy-runtime-0.3.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting grumpy-tools>=0.3.0 (from grumpy-runtime==0.3.0)
...

But then it still fails with error.

...
Building wheels for collected packages: grumpy-runtime
  Building wheel for grumpy-runtime (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/bin/python /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpRJ3HDO                                                                                                                   
       cwd: /tmp/pip-req-build-DJvU2j                                                                                                                                   
  Complete output (757 lines):                                                                                                                                          
  running bdist_wheel                                                                                                                                                   
  running build                                                                                                                                                         
...
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/pydeps", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  Re-executing[2]: make run --debug=bjm -r                                                                                                                              
  GNU Make 4.2.1                                                                                                                                                        
  Built for x86_64-redhat-linux-gnu                                            
...
      Successfully remade target file 'build/src/grumpy/type.go'.                                                                                                       
    Must remake target 'build/pkg/linux_amd64/grumpy.a'.                                                                                                                
    Successfully remade target file 'build/pkg/linux_amd64/grumpy.a'.                                                                                                   
     File 'build/pkg/linux_amd64/__python__/os.a' does not exist.                                                                                                       
       File 'build/src/__python__/os/module.go' does not exist.                                                                                                         
      Must remake target 'build/src/__python__/os/module.go'.                                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/tmp/pip-req-build-DJvU2j/build/bin/grumpc", line 5, in <module>                                                                                              
      from grumpy_tools import cli                                                                                                                                      
  ImportError: No module named grumpy_tools                                                                                                                             
  make: *** [build/stdlib.mk:2181: build/src/__python__/os/module.go] Error 1                                                                                           
  Traceback (most recent call last):                                                                                                                                    
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>                  
      main()                                                                                                                                                            
...
    File "/usr/lib64/python2.7/subprocess.py", line 190, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['echo "print \'Make Runtime Success\'" | make run --debug=bjm -r']' returned non-zero exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for grumpy-runtime
  Running setup.py clean for grumpy-runtime
Failed to build grumpy-runtime
ERROR: Could not build wheels for grumpy-runtime which use PEP 517 and cannot be installed directly

alanjds · 2019-10-03T18:48:33Z

Thanks for your contribution.

The tests are not passing with Go=stable. Would you mind do also fix this before the merge?

abitrolly · 2019-10-04T02:07:06Z

Looks like Python build environment that is set for setup.py is not passed by subprocess to make. Or perhaps make strips the environment. Could also be the problem with shebangs pypa/setuptools#847

abitrolly · 2019-10-05T04:08:18Z

sys.path in setup.py

  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages/pip/_vendor/pep517
  /tmp/pip-build-env-OE7ua4/site
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /tmp/pip-build-env-OE7ua4/overlay/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/overlay/lib64/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib/python2.7/site-packages
  /tmp/pip-build-env-OE7ua4/normal/lib64/python2.7/site-packages

sys.path in subprocess.check_call with shell=True.

  /tmp/pip-req-build-nK1nBr/build/bin
  /tmp/pip-req-build-nK1nBr/build/lib/python2.7/site-packages
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python27.zip
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/plat-linux2
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-tk
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-old
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/lib-dynload
  /usr/lib64/python2.7
  /usr/lib/python2.7
  /home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib/python2.7/site-packages

/tmp/pip-build-env-OE7ua4 with installed grumpy-tools from build-system requires is gone, and /tmp/pip-req-build-nK1nBr without them is present.

abitrolly · 2019-10-05T10:07:15Z

Reported the issue upstream in setuptools. Because build environment is not a proper virtualenv or venv, and setup.py is executed inprocess, the sys.path is not passed to subprocess and it picks up parent pip environment.

The workaround for now is to serialize and pass sys.path to subprocess explicitly as PYTHONPATH environment variable. It will probably mix with virtualenv from pip and if/when upstream fixes that, the workaround should be removed.

abitrolly · 2019-10-05T10:22:57Z

Passing sys.path as PYTHONPATH fails.

  Reading makefiles...                                                             
  Traceback (most recent call last):                                               
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 791, in <module>                                                      
      main()                                                                       
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 781, in main                                                          
      execsitecustomize()                                                          
    File "/home/anatoli/.local/share/virtualenvs/grumpy-PkJvE47S/lib64/python2.7/site.py", line 553, in execsitecustomize                                             
      import sitecustomize                                                         
    File "/tmp/pip-build-env-FWZ6LX/site/sitecustomize.py", line 22, in <module>   
      assert not path in sys.path

abitrolly · 2019-10-05T10:38:21Z

Trying to forcefully inject just a single build path /tmp/pip-build-env-tqb_4I/normal/lib64/python2.7/site-packages doesn't work. It passed as PYTHONPATH to make, but not available when pydeps is run.

Maybe parent vritualenv is killing it.

abitrolly · 2019-10-10T18:34:56Z

@alanjds ready for review. Install works ok.

$ git clone -b fix-install [email protected]:abitrolly/grumpy.git
$ cd grumpy/grumpy-runtime-src
$ pipenv --python 2.7
$ pipenv run pip install .

`pip2 install grumpy-runtime` fails with the ERROR grumpyhome#113 (comment) because it relies on `grumpy-tools`. `grumpy-tools` packages is specified in `setup.py` as a dependency, and it doesn't work, because for the build process to succeed, the dependency should already exist at the time `setup.py` is executed. The official workaround is `pyproject.toml` from PEP-518 pypa/pip#2381

Necessary to locate `grumpy-tools` when building wheel for runtime during `pip install`.

alanjds · 2019-12-27T17:48:25Z

Thanks @abitrolly. I bet this was a pain to tackle.

abitrolly · 2019-12-30T10:21:24Z

Took quite a bit a time, indeed.

abitrolly force-pushed the fix-install branch from 0c37061 to 988b953 Compare October 3, 2019 20:16

abitrolly mentioned this pull request Oct 5, 2019

PEP-518 build env is not preserved for backend subprocess pypa/setuptools#1857

Closed

abitrolly added 3 commits October 10, 2019 22:11

setuptools and wheel are not included by default

9693e46

Preserve PYTHONPATH in Makefile

9d26049

Necessary to locate `grumpy-tools` when building wheel for runtime during `pip install`.

abitrolly force-pushed the fix-install branch from 8934362 to 9d26049 Compare October 10, 2019 19:12

alanjds merged commit ce4d69f into grumpyhome:master Dec 27, 2019

abitrolly deleted the fix-install branch December 29, 2019 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install grumpy-tools before building grumpy-runtime #140

Install grumpy-tools before building grumpy-runtime #140

abitrolly commented Oct 3, 2019

abitrolly commented Oct 3, 2019

alanjds commented Oct 3, 2019

abitrolly commented Oct 4, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 10, 2019

alanjds commented Dec 27, 2019

abitrolly commented Dec 30, 2019

Install grumpy-tools before building grumpy-runtime #140

Install grumpy-tools before building grumpy-runtime #140

Conversation

abitrolly commented Oct 3, 2019

abitrolly commented Oct 3, 2019

alanjds commented Oct 3, 2019

abitrolly commented Oct 4, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 5, 2019

abitrolly commented Oct 10, 2019

alanjds commented Dec 27, 2019

abitrolly commented Dec 30, 2019