General infrastructure for packaging Python packages that use PyO3 / maturin to build native Rust extensions #83614

glittershark · 2020-03-28T18:51:16Z

I've encountered two separate (not currently provided by nixpkgs) python packages that are built with maturin in the last week - orjson and tokenizers - and I can only imagine both that there are more out there, and that more are coming. I have orjson packaged very jankily by building it as a Rust crate directly using naersk, then stripping off the env vars from the resulting derivation and throwing them unceremoniously into buildPythonPackage. That works, but not in a way that I'd even dream of submitting it to nixpkgs - now that I've encountered my second package built using maturin it really is beginning to feel like this is something that buildPythonPackage should "just" handle transparently, the way it does Python packages with C extensions built-in. Anyone have any thoughts?

The text was updated successfully, but these errors were encountered:

FRidh · 2020-03-28T20:31:38Z

In the Python world it is common that a back-end or build tool (often setuptools but also meson) produces a wheel, and a front-end (typically pip) installs it. buildPythonPackage does this using hooks.

I'm unfamiliar with the packages in question. I suppose it's some extension modules, and then some Python code. Ideally a build tool is used that can build a wheel. If not, it indeed becomes hacky.

FRidh · 2020-03-28T20:33:49Z

So orjson has a pyproject.toml. This is the new way for Python packaging. buildPythonPackage supports, just pass format = "pyproject";. You then need to include this maturin in nativeBuildInputs.

bbigras · 2020-06-25T16:55:19Z

@FRidh any ideas why I get ModuleNotFoundError: No module named 'maturin' ?

Should I try to add it to PATH?

EDIT: feel free to ping me on #nixos if you want me to test anything.

default.nix

let
  nixpkgs = import <nixpkgs> {};
in
nixpkgs.python3Packages.buildPythonPackage rec {
  name = "something-${version}";
  version = "0.0.1";
  format = "pyproject";

  nativeBuildInputs = [ nixpkgs.maturin ];

  src = ./.;
}

pyproject.toml

[build-system]
requires = ["maturin"]
build-backend = "maturin"

log

❯ nix-build .
these derivations will be built:
  /nix/store/4q9c2gja115rjyricl05x47v8c8gg6d8-python3.8-something-0.0.1.drv
building '/nix/store/4q9c2gja115rjyricl05x47v8c8gg6d8-python3.8-something-0.0.1.drv'...
Sourcing python-recompile-bytecode-hook.sh
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
unpacking sources
unpacking source archive /nix/store/9b60bppkmhghwvfjwag5z2lm5xvxhvff-maturin
source root is maturin
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file maturin/pyproject.toml
patching sources
configuring
no configure script, doing nothing
building
Executing pipBuildPhase
Creating a wheel...
WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /build/maturin
    Preparing wheel metadata ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 159, in run
    requirement_set = resolver.resolve(
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 550, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 535, in _generate_metadata
    return generate_metadata(
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 176, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/nix/store/8cxkx34jv9jbwxzb5aix53f1bjrpy6aj-python3.8-pip-20.1.1/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/nix/store/jli50q24fvik6pfsnjs1jd90g3dy4p6y-python3-3.8.3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'maturin'


builder for '/nix/store/4q9c2gja115rjyricl05x47v8c8gg6d8-python3.8-something-0.0.1.drv' failed with exit code 2
error: build of '/nix/store/4q9c2gja115rjyricl05x47v8c8gg6d8-python3.8-something-0.0.1.drv' failed

martinetd · 2020-10-26T12:22:43Z

Hi @bbigras.
Have you tried again since then?

I'm working on anki update (#78449), and since there is a Cargo.toml/Cargo.lock I found it easier to package that with buildRustPackage which does all the heavy lifting on the rust side.
The final product is a *-cp38-cp38-manylinux1_x86_64.whl file (that's a wheel?) so definitely a python thing, not sure what I'm missing out by not using buildPythonPackage on that front... It's for a pythonApplication which aren't gated by python subversion so I don't really care about the parallel availability within each python namespace but there might be other problems?

Here's what it (currently) looks like: https://github.com/martinetd/nixpkgs/blob/anki/pkgs/games/anki/rspy/default.nix

Now I'm finally done with that, I just noticed I actually need orjson next, so if I publish my work on anki so our interests might coincide a bit.
I'm not sure how clean that is but for now I'll just try doing something similar for it and adding it to pkgs/top-level/python-packages.nix as a rustPackage.. Let's see how that fares and how I can reuse that.

Comments/yells/whatever welcome -- I'll probably open a PR just for orjson to test the water :P

martinetd · 2020-10-26T12:59:12Z

gah, orjson requires rust nightly! I don't think we can package anything using nightly? And the common workarounds I found don't work... ugh.. I guess it's about time I gave up on an up to date version of anki as well -_-

Sorry for the noise, keep going, nothing to see here!

FRidh · 2020-10-26T15:01:57Z

Had a go at this in #101771.

bbigras · 2020-10-28T23:43:37Z

Have you tried again since then?

I still have the same message with the same default.nix and pyproject.toml, without any python files. I wonder if that could be the cause.

primeos · 2021-02-08T14:39:11Z

I haven't completely analyzed the situation yet but this will likely be required for Nixpkgs soon as python3Packages.cryptography just introduced a Rust dependency in version 3.4: https://cryptography.io/en/latest/changelog.html#v3-4 (though it looks like we can temporarily disable it for 3.4.x: #112402, https://github.com/pyca/cryptography/blob/3.4.1/setup.py#L45)

Unfortunately it also looks like it needs to fetch some Rust dependencies which might be pretty tricky in Nixpkgs:

I'm not yet familiar with PyO3 and maturin but at least PyO3 and PEP 517 seem to be relevant for Cryptography.

SuperSandro2000 · 2021-02-08T20:12:19Z

There is also #108447

FRidh · 2021-02-08T20:47:03Z

Upstream cryptography issue pyca/cryptography#5771.

setuptools-rust is problematic. Not only does it want to use cargo to build rust, which is fine, it also wants to fetch dependencies. Builds should not fetch dependencies.

FRidh · 2021-02-08T21:05:10Z

Opened a topic on the packaging forum to see how other distributors handle this.

FRidh · 2021-02-08T21:19:52Z

Example from @danieldk packaging a Python package using maturin https://github.com/danieldk/nix-packages/blob/81b8444e6864c546cd8f41f393d65fe693c27ca0/pkgs/python-modules/alpino-tokenizer/default.nix#L13

If the wheel would be moved to dist/ in the buildPhase, then pipInstallHook could be used.

danieldk · 2021-02-09T06:01:54Z

setuptools-rust is problematic. Not only does it want to use cargo to build rust, which is fine, it also wants to fetch dependencies. Builds should not fetch dependencies.

tokenizers switched from maturin to setuptools-rust. We currently use the same approach as with maturin-based packages (let buildRustPackage do the fetching + set up the rust environment):

nixpkgs/pkgs/development/python-modules/tokenizers/default.nix

Line 52 in 50d3401

in rustPlatform.buildRustPackage rec {

danieldk · 2021-02-09T08:59:47Z

I have made the cargo vendor handling a hook and I can now build the tokenizers package with buildPythonPackage and without any of the Rust platform stuff. I have to clean it up a bit and integrate the hook in the regular buildRustPackage as well and do a PR.

FRidh · 2021-02-09T09:02:06Z

Great, thank you @danieldk !

danieldk · 2021-02-09T10:47:55Z

Draft PR: #112500

FRidh · 2021-03-13T16:25:01Z

Closing this as done.

glittershark added the 0.kind: packaging request Request for a new package to be added label Mar 28, 2020

FRidh added 6.topic: python 6.topic: rust labels Mar 28, 2020

FRidh mentioned this issue Oct 26, 2020

WIP: python3.pkgs.maturin: PEP 517 build backend #101771

Closed

10 tasks

FRidh self-assigned this Feb 8, 2021

FRidh mentioned this issue Feb 8, 2021

python3Packages.cryptography: 3.3.2 -> 3.4.2 #112402

Closed

10 tasks

FRidh mentioned this issue Feb 8, 2021

buildRustPackage: reimplement as hooks #112438

Open

FRidh mentioned this issue Feb 9, 2021

buildRustPackage: factor out setting up .cargo/config to cargoSetupHook #112500

Merged

10 tasks

This was referenced Feb 11, 2021

buildRustPackage: factor out setting up cargoBuildHook, add maturinBuildHook #112804

Merged

buildRustPackage: factor out cargoCheckHook and cargoInstallHook #113193

Merged

FRidh closed this as completed Mar 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General infrastructure for packaging Python packages that use PyO3 / maturin to build native Rust extensions #83614

General infrastructure for packaging Python packages that use PyO3 / maturin to build native Rust extensions #83614

glittershark commented Mar 28, 2020

FRidh commented Mar 28, 2020

FRidh commented Mar 28, 2020

bbigras commented Jun 25, 2020 •

edited

Loading

martinetd commented Oct 26, 2020

martinetd commented Oct 26, 2020

FRidh commented Oct 26, 2020 •

edited

Loading

bbigras commented Oct 28, 2020

primeos commented Feb 8, 2021

SuperSandro2000 commented Feb 8, 2021

FRidh commented Feb 8, 2021

FRidh commented Feb 8, 2021

FRidh commented Feb 8, 2021 •

edited

Loading

danieldk commented Feb 9, 2021 •

edited

Loading

danieldk commented Feb 9, 2021

FRidh commented Feb 9, 2021

danieldk commented Feb 9, 2021

FRidh commented Mar 13, 2021

General infrastructure for packaging Python packages that use PyO3 / maturin to build native Rust extensions #83614

General infrastructure for packaging Python packages that use PyO3 / maturin to build native Rust extensions #83614

Comments

glittershark commented Mar 28, 2020

FRidh commented Mar 28, 2020

FRidh commented Mar 28, 2020

bbigras commented Jun 25, 2020 • edited Loading

martinetd commented Oct 26, 2020

martinetd commented Oct 26, 2020

FRidh commented Oct 26, 2020 • edited Loading

bbigras commented Oct 28, 2020

primeos commented Feb 8, 2021

SuperSandro2000 commented Feb 8, 2021

FRidh commented Feb 8, 2021

FRidh commented Feb 8, 2021

FRidh commented Feb 8, 2021 • edited Loading

danieldk commented Feb 9, 2021 • edited Loading

danieldk commented Feb 9, 2021

FRidh commented Feb 9, 2021

danieldk commented Feb 9, 2021

FRidh commented Mar 13, 2021

bbigras commented Jun 25, 2020 •

edited

Loading

FRidh commented Oct 26, 2020 •

edited

Loading

FRidh commented Feb 8, 2021 •

edited

Loading

danieldk commented Feb 9, 2021 •

edited

Loading