Skip to content

Commit

Permalink
Write up documentation.
Browse files Browse the repository at this point in the history
  • Loading branch information
bwoodsend committed Sep 24, 2023
1 parent 2537c26 commit 912b41d
Show file tree
Hide file tree
Showing 31 changed files with 1,799 additions and 154 deletions.
100 changes: 91 additions & 9 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,104 @@
Welcome to polycotylus!
=======================

.. image::
https://img.shields.io/pypi/pyversions/polycotylus?label=Python&color=%23184159
:alt: PyPI version
:target: https://pypi.org/project/polycotylus/

Convert Python packages into native Linux distribution packages.

`MIT license <https://github.com/bwoodsend/polycotylus/blob/master/LICENSE>`_
`PyPI <https://pypi.org/project/polycotylus/>`_
P̶y̶P̶I
`Documentation <https://polycotylus.readthedocs.io/>`_
D̶o̶c̶u̶m̶e̶n̶t̶a̶t̶i̶o̶n
`Source code <https://github.com/bwoodsend/polycotylus>`_
`Bug reports <https://github.com/bwoodsend/polycotylus/issues>`_
`Support <https://github.com/bwoodsend/polycotylus/discussions>`_

Polycotylus converts Python packages into native Linux distribution packages
such as RPMs or APKs. It builds on each target Linux distribution (thus dodging
the usual Linux nightmare that is ABI compatibility) and uses each
distribution's packaging tool. In the process, it produces a build script which,
if your project is open source, can be submitted upstream so that your package
will become available on official package repositories (although please don't do
this yet – I want to get some form of review from the distribution maintainers
first).

Polycotylus uses Docker to virtualize each Linux distribution and Qemu to
virtualize almost any architecture meaning that you can build for any supported
distribution or architecture from a single machine. You can even build on
Windows or macOS thanks to Docker Desktop. You can also build apps for Linux
phones: running ``polycotylus manjaro --architecture aarch64`` will build an app
installable on a phone running Manjaro or ``polycotylus alpine --architecture
aarch64`` will build a `postmarketOS <https://postmarketos.org/>`_ compatible
app.

Unlike PyInstaller, Flatpaks or Snaps, polycotylus does not bundle dependencies
into your packages – rather dependencies are declared as such in the package's
metadata where the end user's system package manager will see and act upon them.
This makes the packages tiny, updates modular and propagation of security
patches for vulnerabilities in your dependencies no longer your problem. Complex
system dependencies such as GStreamer or GTK can be declared in addition to PyPI
packages turning them from packaging nightmares into *just another dependency*.
This approach also solves the standard UNIX question of *should I include libXYZ
in my package* to which the answers *yes* and *no* are often simultaneously
wrong.

Polycotylus doesn't just dump your code into an archive and hope for the best –
it verifies it too! It installs your package into a clean, minimal Docker
container and runs your test suite in it. Unless your test suite is a no-op, it
should be almost impossible to forget a dependency or miss a data file without
finding out immediately.


Supported distributions
.......................

Polycotylus is limited by a hard constraint in that it can not support any
target Linux distribution that does not provide ``setuptools>=61.0`` in its
official package repositories. This unfortunately rules out all of the
*stable*/long term support distributions (which also happen to be the most
popular) currently including all stable branches of Debian, Ubuntu <23.04, SLES,
OpenSUSE Leap and all of the RedHat/CentOS-like distributions par Fedora ≥37.
This just leaves rolling build distributions and distributions with fast release
cycles and up to date package repositories.

============= =====================================
Distributions Supported versions
============= =====================================
Alpine_ 3.17-3.18, edge
Arch_ rolling
Fedora_ 37-38, 39 (pre-release), 40 (rawhide)
Manjaro_ rolling
OpenSUSE_ Tumbleweed only (rolling)
Void_ rolling
============= =====================================

There is a work in progress effort to support Debian's unstable branch and
Ubuntu 23.04 but, given ``debpkg``\ 's unwieldiness, I'm not convinced that I'll
ever finish it. 🙁

.. _Alpine: https://alpinelinux.org/
.. _Arch: https://archlinux.org/
.. _Fedora: https://fedoraproject.org/
.. _Manjaro: https://manjaro.org/
.. _OpenSUSE: https://www.opensuse.org/
.. _Void: https://voidlinux.org/


Development status
..................

This project is not considered production ready. It is not available on PyPI nor
is its documentation on readthedocs. To use this project as it is right now, you
will have to install ``polycotylus`` from version control and build the
documentation from source:

.. code-block:: bash
git clone [email protected]:bwoodsend/polycotylus
cd polycotylus
pip install -e .
pip install -r docs/requirements.txt
cd docs
make html
xdg-open build/html/index.html
2 changes: 1 addition & 1 deletion docs/autobuild
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sphinx-autobuild --open-browser --watch ../polycotylus ./source ./build --ignore "*~"
sphinx-autobuild --open-browser --watch ../polycotylus --watch ../README.rst ./source ./build --ignore "*~"
80 changes: 80 additions & 0 deletions docs/source/alpine.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
=========================
Building for Alpine Linux
=========================

Basic usage::

polycotylus alpine

Supported architectures: ``aarch64 armv7 ppc64le x86 x86_64``

Resist the urge to build for all architectures just because you can – you're
unlikely to find a ``ppc64le`` device with a desktop on it or an ``armv7l``
device with enough processing power to run more than a light service or text
editor.

Alpine will produce up to three packages in one build:

* ``main``: Created unconditionally.

* ``doc``: Contains the license file(s). Created only if the license is
considered *non-standard* (which `polycotylus` derives from the `spdx`
option).

* ``pyc``: Contains pre-compiled Python bytecode (a.k.a. the
``__pycache__/*.pyc`` files). Created for Alpine ``>=3.18`` if
`contains_py_files` is true.

You are recommended to ship all three.


Target Alpine version
.....................

Alpine has a long term support release model. Releases are neither forwards nor
backwards compatible; i.e. to support each variant, you need to build separate
packages for each one. `polycotylus` supports building for Alpine ``>=3.17``.
Specify the target version using one of the lines below. No version specifier
implies the latest released version. ::

polycotylus alpine:3.17
polycotylus alpine:3.18
polycotylus alpine:edge # Unstable branch


Package Signing
...............

Alpine makes signing a mandatory step in building and distributing packages. If
you intend to distribute binary packages (as opposed to submitting
``.polycotylus/alpine/APKBUILD`` to Alpine's official package repositories), you
need to make the public key from the key pair used to do the signing is
available to downloaders of your package.

`polycotylus` generates a set of signing keys on the fly the first time you run
``polycotylus alpine``. These keys are stored in the (currently
non-configurable) location ``~/.abuild`` and persist between builds. Put the key
ending in ``.pub`` somewhere publicly downloadable and instruct users to run
the following **before** installing the package itself::

sudo wget https://your-website.com/downloads/your-public-key.rsa.pub -P /etc/apk/keys/

If you're building packages on CI/CD rather than your personal machine then
import your keys into the build machine before running `polycotylus` – otherwise
each build will be signed with its own unique, random key. This can be done with
something like the following:

#. Locally serialise the contents of your ``~/.abuild`` directory. ::

cd ~
tar cz .abuild/ | base64 -w0

#. Copy/paste the output into your CI provider's secret storage.

#. In your CI/CD jobs, pipe the secret into ``base64 -d | tar xz``.
e.g. On GitHub Actions, you could use:

.. code-block:: yaml
- name: Install Alpine signing keys
run: echo '${{ secrets.ALPINE_SIGNING_KEYS }}' | base64 -d | tar xz -C ~
27 changes: 27 additions & 0 deletions docs/source/arch.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
.. _arch_quirks:

=======================
Building for Arch Linux
=======================

Basic (and only) usage::

polycotylus arch

Supported architectures: ``x86_64``


Lifetime of a build
...................

Arch Linux is a rolling build meaning that new versions of packages containing
ABI incompatibilities (or any other incompatibilities arising from the build and
runtime environments being different) can be released at any time and pinning or
downgrading dependencies is prohibited – you're only option is to rebuild and
re-release. One prominent case of this is the annual increment of the Arch
repositories' Python's minor version. When this happens, the ``site-packages``
directory (e.g. ``/usr/lib/python3.11/site-packages/``) moves so that your
package is no longer findable. Once this happens, your existing built packages
are effectively useless and you need to rebuild and release then encourage your
users to run ``pacman -Syu`` (upgrade all packages) before installing/upgrading
your package in case they still have the previous version of Python installed.
1 change: 0 additions & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@
"sphinx.ext.viewcode",
"sphinx_copybutton",
"sphinx.ext.napoleon",
"sphinx.ext.autosectionlabel",
"sphinx.ext.intersphinx",
"sphinx_rtd_theme_configuration",
"sphinx_inline_tabs",
Expand Down
2 changes: 2 additions & 0 deletions docs/source/dependency-constraints.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _dependency_locking:

======================================
Dependency locking/Reproducible builds
======================================
Expand Down
Loading

0 comments on commit 912b41d

Please sign in to comment.