Automated Initial App Distribution for Users

[its-monotype]

Hi, I want some advice regarding the initial distribution of my app.

I have an S3 bucket with a Tufup repository for my app. When releasing a new version I manually upload new metadata and targets but I'm planning to automate it using GitHub Actions.

I need to add a download page on my website featuring a "download" button for the latest version of my app. The thing I'm struggling with is developing the download API route, how would I know what latest version of the target to download?

Maybe I can write a custom simple comparing function by converting the version to numbers (feels a bit hacky, will not cover all PEP440 and it will require caching cause it's slow) or determine by the last modified date (will break if targets are uploaded in random time order), so it feels like not ideal approach, I think it can be simpler. Also here you mentioned that I can repack my app if I want to distribute it in ".zip", maybe it would be also useful, and to implement it a separate distribution bucket is needed, but how to structure it, if, for example, I'll put there only "latest.zip" then I'll need to remove it when deploying next release and there will be downtime which also feels bad, I don't want to mirror all versions in two distribution and auto-updates buckets cause it's not space effective at all. Another thought is to upload the prepared release to GitHub Releases then I can easily access it using GitHub API (https://github.com/USER/PROJECT/releases/latest/download/package.zip), but how would I determine the version for release (I can't get it from the app config, right)? I guess I'll need to use "git tags" and by using them I can trigger a CI workflow that will create a GitHub release, but when creating a version tag I must be sure that it matches with the app version inside the config.

I'm also thinking about distributing for users some pre-release versions (like beta or SMTH).

If we're using an S3 bucket for distribution another idea would be, on the CI release run, to create inside the app repository a separate JSON file and mark somewhere in it the latest versions for stable and pre-release. Or from Tufup metadata parse targets.json -> signed.targets and get the latest entry key (impossible to detect if it's pre-release?). Or call my API within the CI workflow to create a "release" row in DB where I can specify a link to S3 and mark if it's the latest version or pre-release. Or using GitHub releases I can mark it here as pre-release using built-in functionality. I think the S3 bucket with the Tufup repository only for auto-updates and GitHub Releases for initial distribution to users feels like the way to go (probably it's not possible to have Github Releases working for both cause auto-updates require a "repo" pattern where we have not only metadata but other stuff like keys, Tufup config, etc.), but the only thing that worries me is that I need to make sure that the versions match when creating a "git tag", and whether this release version is correct and matches with actual app target version.

So the problem is that I don’t understand what the generally accepted/best approach to this is, maybe it’s a little outside the Tufup scope, but I’d like to know how you do it guys.

[its-monotype]

@dennisvang Any hints, please?

Also, another question is on the CI release workflow you must have the local Tufup repo locally, namely, download from S3:

• .tufup-repo-config to the root
• keystore to the temp
• metadata and targets (I assume that some previous targets needed to create a patch) to the temp/repository

and change paths in .tufup-repo-config to match local paths for keys_dir and repo_dir (until #78 is merged)

But the size of the targets folder can be extremely large if there are a lot of targets, how do you deal with that, maybe only download the latest target by modified date (but what if they were uploaded to S3 in random time order)? Or cache targets on GitHub Actions somehow, idk maybe use a cached target from the previous release workflow run and then clear the previous and update it with the current (feels overcomplicated or not?)... Then after creating and adding a new bundle, I'll need to upload modified metadata, namely snapshot.json, targets.json, and timestamp.json, correct, or do I miss smth? Besides metadata a newly created target must be also uploaded with a patch, these I can grab by the creation date, right?

Btw after solving this, I have an idea to create an example release workflow for the tufup-example.

[dennisvang]

I need to add a download page on my website featuring a "download" button for the latest version of my app. The thing I'm struggling with is developing the download API route, how would I know what latest version of the target to download?

@its-monotype Many options: One option would be to let your web server check for updates using the tufup.client.Client (only check, not install). Another would be to write some code that lets your web-server parse the filenames in the S3 bucket for version numbers, and determine the latest version that way. Yet another option would be to add a separate file to your S3 bucket during deployment, which always points to the latest release. Or you could have your web-server check your source repo for the latest release number, or ... As I said, many options.

[dennisvang]

Maybe I can write a custom simple comparing ...
...
... and whether this release version is correct and matches with actual app target version.

So the problem is that I don’t understand what the generally accepted/best approach to this is, maybe it’s a little outside the Tufup scope, but I’d like to know how you do it guys.

Initial app distribution is really outside the scope of tufup. There are infinitely many ways to approach this problem, all depending on the specifics of your use case. I'm afraid I cannot give any advice on this subject, other than to have a look at how the problem is solved by apps similar to yours.

For example, look at how Python is distributed, or LibreOffice, or PyCharm, or whatever comes close to your own project.

[dennisvang]

... But the size of the targets folder can be extremely large if there are a lot of targets, how do you deal with that, maybe only download the latest target by modified date (but what if they were uploaded to S3 in random time order)? ...

For tufup to work, targets must have the version as part of the file name, so always check the filename, instead of modified date or whatever.
Note that tufup only supports a linear release cycle, so there can be no branching in the version history.

Only the latest version is required when adding a new target, and then only in order to create a patch. If you want to skip patch creation, use the -s option:

$ tufup targets add -h
tufup version: 0.5.1
usage: tufup targets add [-h] [-s] app_version bundle_dir key_dirs [key_dirs ...]

positional arguments:
  app_version       Application version (PEP440 compliant)
  bundle_dir        Directory containing application bundle.
  key_dirs          Directories to search for private and/or public keys.

options:
  -h, --help        show this help message and exit
  -s, --skip-patch  Skip patch creation.

[its-monotype]

That makes sense, but let's assume we are inside the GitHub actions workflow on stage when we need to create a new target for auto-updates of our app. Besides other Tufup local repo setup steps, we need to download the latest Tufup target from the S3 Tufup remote repository, as you said we need to check the filename to parse the latest version. So we will need some PEP440-compliant parsing function to get the latest version from the filename, or another way?

[dennisvang]

tufup uses packaging.version.Version internally, see packaging docs for usage.

For convenience, you could use tufup's TargetMeta, which takes care of the the filename parsing:

>>> from tufup.common import TargetMeta
>>> t1 = TargetMeta('my-app-1.2.3beta45.tar.gz')
>>> t1.version
<Version('1.2.3b45')>
>>> t2 = TargetMeta('my-app-1.2.3rc0.tar.gz')
>>> t2 > t1
True

Note: valid filenames are defined by this regex pattern:

r'^(?P<name>[\w-]+)-(?P<version>.+)(?P<suffix>\.tar\.gz|\.patch)$'

[dennisvang]

I'm also thinking about distributing for users some pre-release versions (like beta or SMTH).

Just make sure your version specifier complies with PEP440 (see pre-release section), e.g. 1.2.3a.

For the repository side it does not make any difference whether your version represents a final release or a pre-release.
However, the tufup client disregards pre-releases by default, when checking for new updates. From the docstring for check_for_updates():

Final releases are always included. Pre-releases are excluded by default. If pre is specified, pre-releases are included, down to the specified level. Pre-release identifiers follow the PEP440 specification, i.e. 'a', 'b', or 'rc', for alpha, beta, and release candidate, respectively.

So, your app will need to tell the tufup client to look for pre-releases in addition to final releases. For example check_for_updates(pre='a') would include all final releases as well as pre-releases down to the alpha level, i.e. 'a', 'b', and 'rc', whereas e.g. pre='rc' would only include 'rc' pre-releases (and final releases).