Add developer docs about zkg's internal use of directories

doc/developers.rst

@@ -88,3 +88,116 @@ INI File Option            ```name```                      `name`
 ========================== =============================== ===========================
 
 Python API docstrings roughly follow the `Google Style Docstrings`_ format.
+
+Internals
+---------
+
+``zkg``'s view of a package
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``zkg`` maintains copies of a Zeek package in up to four places:
+
+- A long-lived clone of the package's git repo in
+  ``$STATEDIR/clones/package/<name>``. This clone (not its installed version,
+  see below) is ``zkg``'s "authoritative" view of a package.
+
+- A conceptually short-lived clone in ``$STATEDIR/scratch/<name>``, for
+  retrieving information about a package ("short-lived", because access to those
+  copies is momentary -- not because ``zkg`` necessarily cleans up after those
+  copies).
+
+- A "stage". A stage is any place in which ``zkg`` actually installs a package
+  for running in Zeek. This can be the local Zeek installation, or locally below
+  ``$STATEDIR/testing`` when testing a package. So "staging" here does not mean
+  "not yet live"; rather the opposite: it's closer to "any place where the
+  package may actually run". A stage allows customization of many of the
+  directories involved (for internal cloning, installing Zeek scripts and
+  plugins, binary files, etc). Installation into a stage is also where ``zkg``
+  adds its management of ``packages.zeek``, in the stage's scripts directory.
+
+- In ``$STATEDIR/testing/<name>/clones/<names>``, for testing a given package
+  along with any dependencies it might have.
+
+``zkg`` captures state about installed packages in ``$STATEDIR/manifest.json``.
+This does not capture all knowable information about packages, though.
+More on this next.
+
+Directory usage
+~~~~~~~~~~~~~~~
+
+``zkg`` populates its internal state directory (dubbed ``$STATEDIR`` below) with
+several subdirectories.
+
+``$STATEDIR/clones``
+""""""""""""""""""""
+
+This directory keeps git clones of installed packages
+(``$STATEDIR/clones/package/<name>``), packages sources
+(``$STATEDIR/clones/source/<name>``), and package template repositories
+(``$STATEDIR/clones/template/<name>``).
+
+``zkg`` clones the relevant repositories as needed. It can dynamically re-create
+some of these directories as needed, but interfering in that space is not
+recommended. For example, if you remove a clone of an installed package, the
+installation itself will remain live (via the staging mechanism), but ``zkg``
+will no longer be able to refer to all information about the installed package
+(because anything not explicitly captured about the package in
+``$STATEDIR/manifest.json`` is now gone).
+
+Removal of a package (``zkg remove``) removes its clone in ``$STATEDIR/clones``.
+
+``$STATEDIR/scratch``
+"""""""""""""""""""""
+
+When retrieving information on a package that isn't yet installed, or where
+``zkg`` doesn't want to touch the installed code, ``$STATEDIR/scratch/<name>``
+is a clone of the package's git repo at a version (often a git tag) of
+interest. This clone is shallow for any versions that aren't raw SHA-1
+hashes. The information parsed includes the ``zkg.meta`` as well as git
+branch/tag/commit info.
+
+During package installation, ``zkg`` places backups of user-tweakable files into
+``$STATEDIR/scratch/tmpcfg``. ``zkg`` restores these after package installation
+to preserve the user's edits. During package source aggregation, ``zkg`` places
+temporary versions of ``aggregate.meta`` directly into ``$STATEDIR/scratch``.
+
+Creation or unbundling of a package happens via ``$STATEDIR/scratch/bundle``, to
+compose or retrieve information about the bundle. The directory is deleted and
+re-created at the beginning of those operations:
+
+- During bundling, ``zkg`` copies installed package repos from
+  ``$STATEDIR/clones/<name>`` into ``$STATEDIR/scratch/bundle/<name>``, and
+  creates fresh git clones in the ``bundle`` directory for any packages not
+  currently installed. It creates a ``.tar.gz`` of the whole directory,
+  initially in the bundle directory, and moves it to where the user specified.
+
+- During unbundling, ``zkg`` reads the bundle manifest as well as the git repos
+  of the contained packages, and moves the package repos from the scratch space
+  into ``$STATEDIR/clones/package/<name>``.
+
+When installing a package's ``script_dir`` or ``plugin_dir`` into a staging area
+and the source file is a tarfile, ``zkg`` temporarily extracts the tarball into
+``$STATEDIR/scratch/untar/``.
+
+There's little or no cleanup of files in the scratch space after the operations
+creating them complete. ``zkg`` only deletes, then (re-)creates, the directories
+involved upon next use.
+
+When ``zkg`` isn't in the middle of executing any commands, you can always
+delete the scratch space without negatively affecting ``zkg``.
+
+``$STATEDIR/testing``
+"""""""""""""""""""""
+
+When testing a package (during installation, or when explicitly running ``zkg
+test``), ``zkg`` creates a staging area ``$STATEDIR/testing/<name>`` for the
+package under test, clones the package and its dependencies into
+``$STATEDIR/testing/<name>/clones/``, installs them from there into
+``$STATEDIR/testing/<name>``, and then runs the package's ``test_command`` from
+its clone (``$STATEDIR/testing/<name>/clones/<name>``), with an environment set
+such that it finds the installation in the local stage. The stdout and stderr of
+those testing runs is preserved into ``zkg.test_command.stdout`` and
+``zkg.test_command.stderr`` in that directory.
+
+As in ``STATEDIR/scratch``, there's no cleanup, and you can delete the testing
+space as needed after a test run is complete.