pynixify has the following objectives, which other alternatives don't satisfy (at least from my point of view):
- Be usable in big projects: pynixify was initially developed to build a Nix expression for Faraday, a project with lots of dependencies, some of which weren't packaged in Nixpkgs. Most alternatives to this tool didn't work properly because they weren't reusing Nixpkgs packages.
- Reuse Nixpkgs expressions: instead of writing expressions for all transitive dependencies from scratch, take as much as possible from Nixpkgs. This makes it possible to use packages with system dependencies or with complex build steps, such as Pillow.
- Generate human-readable expressions: the generated code is properly formatted and uses Nixpkgs best practices (such as adding package metadata and including test dependencies). This facilitates contributing to Nixpkgs, since the generated expression will be of "Nixpkgs quality". Here is an example of the expression of pynixify itself.
The only supported installation method is using Nix:
$ git clone https://github.com/cript0nauta/pynixify.git
$ cd pynixify
$ nix-env -if .
Lets suppose you want to use pypa's sampleproject. Sadly,
Nixpkgs doesn't provide a python3Packages.sampleproject attribute yet. This
means you'll have to write it. This isn't too difficult, but can be tedious,
especially with projects with many dependencies. Instead, you can use pynixify
to build the expression for you:
$ pynixify sampleproject
Resolving sampleproject
Resolving peppercorn (from PyPIPackage(attr=sampleproject, version=2.0.0))
$ nix-build pynixify/nixpkgs.nix -A python3Packages.sampleproject
/nix/store/fpa32bnl2r5vwdsybrz8bvw7qhcvvik3-python3.7-sampleproject-2.0.0
$ ./result/bin/sample
Call your main application code here
The generated pynixify/nixpkgs.nix file should be used as a
replacement of <nixpkgs>. It includes an overlay with all package definitions
generated in pynixify/packages/. In this case, it will include the definition
for sampleproject.
When you're developing a package, fetching its source from PyPI won't be
useful. It would be better to use a local directory, so the changes you made in
your machine are instantly reflected. When you use the --local option,
pynixify will use the current directory as the package source, making package
development easier:
$ git clone https://github.com/pypa/sampleproject.git
$ cd sampleproject/
$ pynixify --local sampleproject
Resolving sampleproject
Resolving peppercorn (from PyPIPackage(attr=sampleproject, version=0.1dev))
$ nix-shell pynixify/nixpkgs.nix -A python3Packages.sampleproject
[nix-shell:/tmp/sampleproject]$ echo " print('Hello from pynixify! ')" >>src/sample/__init__.py
[nix-shell:/tmp/sampleproject]$ sample
Call your main application code here
Hello from pynixify!
Using both nix-shell and nix-build will use the current directory as
source. The --local option is also useful if you include your pynixify/
directory inside your Git repository. This facilitates setting up a development
environment of your project.
If you include the pynixify/ directory inside your Git repository, it is
recommended to use a pinned version of Nixpkgs. This will improve
reproducibility between different machines. pynixify's --nixpkgs option
makes the generated expression use an exact version of nixpkgs:
$ pynixify sampleproject --nixpkgs https://github.com/NixOS/nixpkgs/archive/748c9e0e3ebc97ea0b46a06295465eff2fb5ef92.tar.gz
$ cat pynixify/nixpkgs.nix
[...]
nixpkgs =
builtins.fetchTarball {
url =
"https://github.com/NixOS/nixpkgs/archive/748c9e0e3ebc97ea0b46a06295465eff2fb5ef92.tar.gz";
sha256 = "158xblbbjv54n9a7b1y88jjjag2w5lb77dqfx0d4z2b32ss0p7mc";
};
[...]
Some Python projects don't have a setup.py file to indicate how should they
be built. In most cases, they just have a requirements.txt file indicating
which packages need to be installed. If this is the case of your project, you
can use the pynixify/shell.nix file to setup a virtualenv-like environment
with all requirements installed:
$ pynixify -r requirements.txt
$ nix-shell pynixify/shell.nix
You can also specify which version of Python you want:
$ nix-shell pynixify/shell.nix --argstr python python36
Using pynixify can be a great way to introduce Nix to your team. Instead of
complex Dockerfiles and install scripts, developers can just run nix-shell
and set up a reproducible development environment immediately!
In addition to adding the pynixify/ directory to your Git repo, it is
suggested to manually create a default.nix similar to pynixify's
one (you may also want to pin Nixpkgs). This
provides some advantages over using the pynixify/ files directly:
- Want to build the project? Just run
nix-buildwith no arguments. There is no morenix-build pynixify/nixpkgs.nix -A yourProjectName. - There is no need for an additional
shell.nixfile, sincebuildPythonPackageactivates the development mode automatically when usingnix-shell. Just adefault.nixfile is good enough. - As detailed in the example file, you can override the generated package to add system dependencies in case they're needed .
For a real life example, you can look at Faraday's release.nix file. It uses pynixify's generated definition, overrides a few things and adds instructions to build a Docker image and a Systemd unit for the project.
pynixify tries to reuse Nixpkgs packages whenever possible. Although this is mostly consireded a good thing, it has some drawbacks:
- You won't be using the latest PyPI packages, but the versions available in
Nixpkgs, which can be older. If you must a newer version of a library,
consider adding
>=to its requirement. If the library has a standard build mechanism, that will work fine. Otherwise, it is recommended to manually change the library's version in the Nixpkgs repository. - Your requirements file shouldn't be the output of
pip freezebecause it has==in all requirements. Therefore, it makes reusing Nixpkgs packages harder. You shouldn't treat your requirements file as a lockfile, but as an abstract definition indicating the minimum and maximum versions of each library. Keep in mind that you don't need a lockfile: you can just pin Nixpkgs to have something with the reproducibility of lockfiles. Or even better than it, since Nix also tracks the system dependencies of each library!
# A bad requirements.txt file
certifi==2020.6.20
chardet==3.0.4
idna==2.10
requests==2.24.0
tqdm==4.48.2
urllib3==1.25.10
# A great requirements.txt file
requests
tqdm>=4.47.0
- mach-nix: A good alternative to pynixify. It is more focused on non-Nix users, while pynixify targets users with experience using Nix, and wanting to contribute to its ecosystem via Nixpkgs Pull requests.
- nixpkgs-pytools: This library also targets creating packages for nixpkgs. Unlike pynixify, it doesn't automatically create expressions for the missing dependencies, and the generated expressions require manual tweaking before being valid Nix. pynixify would be located in the middle between nixpkgs-pytools and mach-nix.
- poetry2nix: If you use poetry, this can be a good alternative too. If you're interested, see this great blogpost about it. Most of the concepts in it apply to pynixify too!
- pypi2nix: This one is abandoned and never worked for me. I wouldn't recommend it.
PRs and issues describing bugs packaging some libraries are more than welcome!
To setup a development environment, just clone this repo and run nix shell.
pynixify has three different test suites:
- unit tests, which are designed to be fast and isolated
- "acceptance" tests which require network access and are slower
- bats end to end tests, which test running the
pynixifycommand with different options
You can run all three of them inside nix-shell:
$ pytest tests/ acceptance_tests/
$ bats acceptance_tests/test_command.sh
They're also run on each push using GitHub actions.