Notes from my talk

2021/2021-08-09/ldo/ctypes example.ipynb

@@ -0,0 +1,206 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Load the shareable library (note versioned file from runtime package, not development package):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import ctypes as ct\n",
+    "\n",
+    "cairo = ct.cdll.LoadLibrary(\"libcairo.so.2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note the use of the explicitly-versioned library name `libcairo.so.2` instead of `libcairo.so`; latter comes from development package, former from run-time package. Development package should only be needed for building compiled code (e.g. in C) against the library; for users who only need to use your module to run Python code, the run-time package should be sufficient."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**API Versus ABI** The versioning of the shared library file is to deal with incompatible changes in generated code in client programs. For example, a structure layout might change, without requiring any changes to client _source_ code: but once the client app is compiled against the new layout, it will not work against the old one, and vice versa. So existing code cannot load the new version of the library, it must continue to load the old one, while newly-built code uses the new library."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Representation of [`cairo_matrix_t`](https://www.cairographics.org/manual/cairo-cairo-matrix-t.html#cairo-matrix-t) (also add a `__repr__` for easy debugging):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class matrix_t(ct.Structure) :\n",
+    "    _fields_ = \\\n",
+    "        [\n",
+    "            (\"xx\", ct.c_double),\n",
+    "            (\"yx\", ct.c_double),\n",
+    "            (\"xy\", ct.c_double),\n",
+    "            (\"yy\", ct.c_double),\n",
+    "            (\"x0\", ct.c_double),\n",
+    "            (\"y0\", ct.c_double),\n",
+    "        ]\n",
+    "\n",
+    "    def __repr__(self) :\n",
+    "        return \"[\" + \", \".join(repr(getattr(self, f[0])) for f in matrix_t._fields_) + \"]\"\n",
+    "    #end __repr__\n",
+    "#end matrix_t"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Define routine prototypes, e.g."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cairo.cairo_matrix_init_identity.restype = None\n",
+    "cairo.cairo_matrix_init_identity.argtypes = (ct.POINTER(matrix_t),)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Example creation of C object:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "m = matrix_t()\n",
+    "m"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Pass to library routine and observe result:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cairo.cairo_matrix_init_identity(ct.byref(m))\n",
+    "m"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Try another routine:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cairo.cairo_matrix_scale.restype = None\n",
+    "cairo.cairo_matrix_scale.argtypes = (ct.POINTER(matrix_t), ct.c_double, ct.c_double)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cairo.cairo_matrix_scale(m, 3, 2)\n",
+    "m"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note how you could pass straight Python numeric expressions for `c_double` args; conversion for such simple types is automatic."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "libc = ct.cdll.LoadLibrary(\"libc.so.6\")\n",
+    "\n",
+    "class FILE(ct.Structure) :\n",
+    "    _fields_ = []\n",
+    "#end FILE\n",
+    "\n",
+    "stderr = ct.POINTER(FILE).in_dll(libc, \"stderr\")\n",
+    "\n",
+    "\n",
+    "libc.fprintf.argtypes = (ct.POINTER(FILE), ct.c_char_p,)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "libc.fprintf(stderr, \"hello, world!\".encode())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

2021/2021-08-09/ldo/notes.txt

@@ -0,0 +1,121 @@
+There is frequently a need in Python code to access libraries for
+various functionality not directly available in Python itself. These
+libraries are often written for use from C or C++ code, not from
+Python, so you need to create some kind of Python wrapper for them.
+One common answer is to write an “extension module” in C or C++, using
+the Python C API <https://docs.python.org/3/c-api/index.html>,
+<https://docs.python.org/3/extending/index.html> to wrap the library
+functionality in Python objects.
+
+However, there is another approach, and that is to create the wrapper
+entirely in Python, using the ctypes
+<https://docs.python.org/3/library/ctypes.html> module. When I first
+started reading about this, I assumed that it was a clunky solution
+that should be used as a last resort -- that it would be preferable
+in terms of usability, efficiency etc to go the extension-module
+route, wherever possible.
+
+However, I gradually found out that this is wrong. ctypes is actually
+a very well-designed module, which can be used to create very
+“Pythonic” wrappers -- that is, they behave as though the library
+directly supports Python objects, with little or no hint that the
+underlying library is actually implemented in C or C++, with no
+thought for Python. In other words, using a ctypes wrapper can quite a
+pleasant experience.
+
+As an example, I am going to look at the Cairo graphics API
+<https://www.cairographics.org/>, a powerful and versatile 2D graphics
+engine. This has long had an “official” (?) Python binding, called
+Pycairo <https://pycairo.readthedocs.io/en/latest/>. This binding is
+not a complete wrapper for the functionality of Cairo: there was no
+upstream development for many years (between 2011 and 2017
+<https://pycairo.readthedocs.io/en/latest/changelog.html>), leaving it
+with many gaps in its functionality. These have been mostly remedied
+now, with the notable exception of fonts: there is still no support
+for FreeType fonts
+<https://www.cairographics.org/manual/cairo-FreeType-Fonts.html> or
+user fonts
+<https://www.cairographics.org/manual/cairo-User-Fonts.html>, and
+another thing that continues to annoy me: no option to pass arguments
+by keyword!
+
+During this hiatus, I spent some time looking at filling in some of
+these gaps, before deciding that it would be less effort to start
+again from scratch.
+
+** Walkthrough
+    example of ctypes calls, using accompanying ctypes_example.ipynb
+**
+
+So I created Qahirah <https://github.com/ldo/qahirah>, a pure-Python
+binding for Cairo, using ctypes.
+
+For comparison, Pycairo has about 10,000 lines of C code, while
+qahirah.py is just about 8000 lines of Python code. The main omissions
+from the latter are support for on-screen drawing through
+platform-specific GUIs, but it has essentially everything else,
+including all the font support. Plus I was able to add high-level
+Python objects for Vector, Matrix, Colour and Path types. And being
+pure Python, you get keyword arguments for free. It can also take
+advantage of my pure-Python bindings for FreeType
+<https://github.com/ldo/python_freetype/> and Fontconfig
+<https://github.com/ldo/python_fontconfig/> if these are installed.
+
+The way I like to think of Qahirah is: it’s what I think a graphics
+API should look like if it was designed specifically for Python.
+
+What are the supposed advantages of implementing
+an extension module in compiled C/C++ code? As far as
+I can tell, they are
+* You can keep the insides of the module really private,
+  whereas a Python module will always have its insides
+  accessible from outside.
+* Compiled code is more efficient than interpreted Python
+  code.
+
+I don’t see the first advantage as really much of an advantage. Guido
+van Rossum, when asked why Python does not have visibility controls on
+its modules and classes, replied “we’re all consenting adults here”.
+Also, because a Python wrapper using ctypes does not completely lock
+down its internal interface to the C library, it is possible, without
+a great deal of effort, for users to resort to their own lower-level
+accesses if they feel they need to, to handle situations that the
+implementor of the wrapper did not envisage. This means
+flexibility.
+
+As for the efficiency issue, there may be libraries where it does
+matter, but interfacing to Cairo is not one of them. Rendering graphics
+means doing a lot of low-level pixel manipulation. So a lot of the CPU
+time is spent inside the Cairo library doing just that, not in
+interfacing to Python.
+
+If you look at the Pycairo docs, the reason they give for not
+supporting the additional font functionality is that “there is no open
+source Python bindings [sic] to FreeType (and fontconfig) that
+provides a C API”. Basically, they want another extension module for
+Python that provides an API that allows the Pycairo extension module
+to hook directly into it. They provide such a C API themselves
+<https://pycairo.readthedocs.io/en/latest/pycairo_c_api.html> to allow
+other extension modules to interface with them. But it seems to me
+this is an overcomplicated and unnecessary way of doing things.
+
+** Walkthrough
+    comparison of doing Cairo calls: C versus Pycairo versus
+    Qahirah, using accompanying spirotri programs
+**
+
+Limitations of ctypes? only two major ones that I have found:
+* Cannot pass a C struct by value, only by address
+  (e.g. in Fontconfig, args of Value type -- thankfully not
+  essential, there were alternative type-specific calls I was able
+  to use)
+* Cannot handle setjmp/longjmp -- required by libpng for
+  its exception handling, which provides no other options
+  for error notification besides setjmp/longjmp or aborting
+  the program. This one leaves me stuck for now.
+Another minor one: entry points have to have a fixed argument
+signature, so it doesn’t seem to handle varargs routines well.
+This hasn’t been a big deal for me for a couple of reasons:
+* I rarely need to call such routines
+* I figured out a hack to make copies of the entry-point object so
+  each instance can be assigned a different argument signature.