GCC fails when trying to install reppy through pip #132

dcfreire · 2022-04-29T20:45:09Z

I cannot install reppy in either python 3.8 or 3.10.

# pip3.8 install reppy                                                                                                                                                                  
Collecting reppy
  Using cached reppy-0.4.14.tar.gz (93 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: cachetools in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from reppy) (5.0.0)
Requirement already satisfied: python-dateutil!=2.0,>=1.5 in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from reppy) (2.8.2)
Requirement already satisfied: requests in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from reppy) (2.27.1)
Requirement already satisfied: six in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from reppy) (1.16.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from requests->reppy) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from requests->reppy) (1.26.9)
Requirement already satisfied: idna<4,>=2.5 in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from requests->reppy) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages (from requests->reppy) (2.0.12)
Building wheels for collected packages: reppy
  Building wheel for reppy (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/shi/.virtualenvs/ir-1/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-40ydxajn
       cwd: /tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/
  Complete output (42 lines):
  Building from C++
  /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
    warnings.warn(
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/reppy
  copying reppy/ttl.py -> build/lib.linux-x86_64-3.8/reppy
  copying reppy/exceptions.py -> build/lib.linux-x86_64-3.8/reppy
  copying reppy/util.py -> build/lib.linux-x86_64-3.8/reppy
  copying reppy/__init__.py -> build/lib.linux-x86_64-3.8/reppy
  creating build/lib.linux-x86_64-3.8/reppy/cache
  copying reppy/cache/policy.py -> build/lib.linux-x86_64-3.8/reppy/cache
  copying reppy/cache/__init__.py -> build/lib.linux-x86_64-3.8/reppy/cache
  running build_ext
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/reppy
  creating build/temp.linux-x86_64-3.8/reppy/rep-cpp
  creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps
  creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp
  creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp/src
  creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/src
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DOPENSSL_NO_SSL2 -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fPIC -Ireppy/rep-cpp/include -Ireppy/rep-cpp/deps/url-cpp/include -I/home/shi/.virtualenvs/ir-1/include -I/usr/include/python3.8 -c reppy/rep-cpp/deps/url-cpp/src/psl.cpp -o build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp/src/psl.o -std=c++11
  In file included from reppy/rep-cpp/deps/url-cpp/src/psl.cpp:7:
  reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:54: error: ‘numeric_limits’ is not a member of ‘std’
     56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
        |                                                      ^~~~~~~~~~~~~~
  reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:82: error: expected primary-expression before ‘>’ token
     56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
        |                                                                                  ^
  reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:85: error: ‘::max’ has not been declared; did you mean ‘std::max’?
     56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
        |                                                                                     ^~~
        |                                                                                     std::max
  In file included from /usr/include/c++/11.2.0/algorithm:62,
                   from reppy/rep-cpp/deps/url-cpp/src/psl.cpp:1:
  /usr/include/c++/11.2.0/bits/stl_algo.h:3467:5: note: ‘std::max’ declared here
   3467 |     max(initializer_list<_Tp> __l, _Compare __comp)
        |     ^~~
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for reppy
  Running setup.py clean for reppy
Failed to build reppy
Installing collected packages: reppy
    Running setup.py install for reppy ... error
    ERROR: Command errored out with exit status 1:
     command: /home/shi/.virtualenvs/ir-1/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rsopie2y/install-record.txt --single-version-externally-managed --compile --install-headers /home/shi/.virtualenvs/ir-1/include/site/python3.8/reppy
         cwd: /tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/
    Complete output (44 lines):
    Building from C++
    /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
      warnings.warn(
    running install
    /home/shi/.virtualenvs/ir-1/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/reppy
    copying reppy/ttl.py -> build/lib.linux-x86_64-3.8/reppy
    copying reppy/exceptions.py -> build/lib.linux-x86_64-3.8/reppy
    copying reppy/util.py -> build/lib.linux-x86_64-3.8/reppy
    copying reppy/__init__.py -> build/lib.linux-x86_64-3.8/reppy
    creating build/lib.linux-x86_64-3.8/reppy/cache
    copying reppy/cache/policy.py -> build/lib.linux-x86_64-3.8/reppy/cache
    copying reppy/cache/__init__.py -> build/lib.linux-x86_64-3.8/reppy/cache
    running build_ext
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/reppy
    creating build/temp.linux-x86_64-3.8/reppy/rep-cpp
    creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps
    creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp
    creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp/src
    creating build/temp.linux-x86_64-3.8/reppy/rep-cpp/src
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DOPENSSL_NO_SSL2 -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fPIC -Ireppy/rep-cpp/include -Ireppy/rep-cpp/deps/url-cpp/include -I/home/shi/.virtualenvs/ir-1/include -I/usr/include/python3.8 -c reppy/rep-cpp/deps/url-cpp/src/psl.cpp -o build/temp.linux-x86_64-3.8/reppy/rep-cpp/deps/url-cpp/src/psl.o -std=c++11
    In file included from reppy/rep-cpp/deps/url-cpp/src/psl.cpp:7:
    reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:54: error: ‘numeric_limits’ is not a member of ‘std’
       56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
          |                                                      ^~~~~~~~~~~~~~
    reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:82: error: expected primary-expression before ‘>’ token
       56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
          |                                                                                  ^
    reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:85: error: ‘::max’ has not been declared; did you mean ‘std::max’?
       56 |         const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
          |                                                                                     ^~~
          |                                                                                     std::max
    In file included from /usr/include/c++/11.2.0/algorithm:62,
                     from reppy/rep-cpp/deps/url-cpp/src/psl.cpp:1:
    /usr/include/c++/11.2.0/bits/stl_algo.h:3467:5: note: ‘std::max’ declared here
     3467 |     max(initializer_list<_Tp> __l, _Compare __comp)
          |     ^~~
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/shi/.virtualenvs/ir-1/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b0oy_bsz/reppy_b67f12bbf68043dab8f5014f214e8614/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rsopie2y/install-record.txt --single-version-externally-managed --compile --install-headers /home/shi/.virtualenvs/ir-1/include/site/python3.8/reppy Check the logs for full command output.

The text was updated successfully, but these errors were encountered:

dcfreire · 2022-05-04T21:43:53Z

I built reppy from source for now. The issue is the non-inclusion of the <limits> library on punycode.h, which is a header file on the url-cpp dependency. (As per the error message)

Gallaecio · 2022-06-03T07:18:42Z

Workaround: use CLang:

CC=clang pip install reppy

wumpus · 2022-06-03T07:28:07Z

The usual python solution is to have a wheel, so people installing don't have to think about which compiler to use.

Gallaecio · 2022-07-25T09:35:01Z

Interestingly, using CLang does not work for me when using Python 2.7, but it works with Python 3.8.

diyanqi · 2022-08-05T13:01:12Z

find a solution:
add following code in the front of the file reppy/rep-cpp/deps/url-cpp/include/punycode.h :

#include <stdexcept>
#include <limits>

mfhepp · 2022-08-26T10:11:05Z

@diyanqi
Thanks - IMO this would be better fixed in https://github.com/seomoz/url-cpp, i.e. here

https://github.com/seomoz/url-cpp/blob/master/include/punycode.h

Then, as this is a Git submodule in here, the frozen version in Reppy would need to be updated.

But I think the better way is retiring Reppy entirely and using the official Python module instead:

https://docs.python.org/3.10/library/urllib.robotparser.html#module-urllib.robotparser

Gallaecio · 2022-08-26T10:34:25Z

There are quite a few reasons to prefer Reppy.

mfhepp · 2022-08-26T12:04:39Z

@Gallaecio Thanks for the good pointer!

However, what shall we do with a broken module that is no longer maintained? Is there an official or at least stable fork of Reppy?

Also, in the age of large-scale Supply Chain Attacks on Open-Source libraries, I think many developers will prefer a standard library module than one that is no longer maintained and has outdated dependencies. At least the frozen version for requests has quite some vulnerabilities (not super-critical ones, but still).

Gallaecio · 2023-01-02T13:08:29Z

CLang does not work for me on GitHub Actions either, same error as with GCC.

clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Ireppy/rep-cpp/include -Ireppy/rep-cpp/deps/url-cpp/include -I/home/runner/work/scrapy/scrapy/.tox/pylint/include -I/opt/hostedtoolcache/Python/3.8.15/x64/include/python3.8 -c reppy/rep-cpp/deps/url-cpp/src/psl.cpp -o build/temp.linux-x86_64-cpython-38/reppy/rep-cpp/deps/url-cpp/src/psl.o -std=c++11
      In file included from reppy/rep-cpp/deps/url-cpp/src/psl.cpp:7:
      reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:54: error: no member named 'numeric_limits' in namespace 'std'
              const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
                                                      ~~~~~^
      reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:69: error: unexpected type name 'punycode_uint': expected expression
              const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
                                                                          ^
      reppy/rep-cpp/deps/url-cpp/include/punycode.h:56:85: error: no member named 'max' in the global namespace
              const punycode_uint MAX_PUNYCODE_UINT = std::numeric_limits<punycode_uint>::max();
                                                                                        ~~^
      3 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1

#132 (comment) seems like the only solution, but I don‘t think there is a clean way to patch like that while still using pip to install.

Gallaecio mentioned this issue Jun 2, 2022

Deprecate Reppy support scrapy/scrapy#5230

Closed

rivermont mentioned this issue Apr 17, 2023

Update for Python 3.9+, deprecate Reppy dependency rivermont/spidy#89

Open

3 tasks

micahcochran mentioned this issue Sep 25, 2023

Installation problems with reppy micahcochran/recipe-crawler#10

Closed

wRAR mentioned this issue Oct 17, 2023

Deprecate ReppyRobotParser. scrapy/scrapy#6099

Merged

This was referenced Jan 12, 2024

Picked up <limits> include bugfix in url-cpp to address compile errors on newer platforms seomoz/rep-cpp#42

Merged

Point to new rep-cpp, which points to new url-cpp, fixing <limits> #133

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCC fails when trying to install reppy through pip #132

GCC fails when trying to install reppy through pip #132

dcfreire commented Apr 29, 2022

dcfreire commented May 4, 2022

Gallaecio commented Jun 3, 2022

wumpus commented Jun 3, 2022

Gallaecio commented Jul 25, 2022

diyanqi commented Aug 5, 2022

mfhepp commented Aug 26, 2022

Gallaecio commented Aug 26, 2022

mfhepp commented Aug 26, 2022

Gallaecio commented Jan 2, 2023 •

edited

Loading

GCC fails when trying to install reppy through pip #132

GCC fails when trying to install reppy through pip #132

Comments

dcfreire commented Apr 29, 2022

dcfreire commented May 4, 2022

Gallaecio commented Jun 3, 2022

wumpus commented Jun 3, 2022

Gallaecio commented Jul 25, 2022

diyanqi commented Aug 5, 2022

mfhepp commented Aug 26, 2022

Gallaecio commented Aug 26, 2022

mfhepp commented Aug 26, 2022

Gallaecio commented Jan 2, 2023 • edited Loading

Gallaecio commented Jan 2, 2023 •

edited

Loading