How does signing actually work?

[Eoic]

I don't understand signing part at all. I read Roles and metadata section in TUF website but I still don't understand the process. When / what / how should I sign metadata files?

This is what I do:

1. Initialize the repository.
2. Build application, version 1.0.0, with pyinstaller (root.json is also included with the application).
3. Add bundle to TUF repository.
4. Start HTTP server to serve repository files.
5. Change something in the application, update version number and build it again (version 1.0.1).
6. Add bundle to TUF repository, which results in patch file creation.
7. Start the application built in step 1 (version 1.0.0).

I use tufup Client to check for updates on start-up. It does not find any and outputs an error message, for example, "Cannot refresh metadata: root was signed by 0/1 keys". Sometimes it's "timestamp" instead of "root". I tried signing root, targets, timestamp with command, for example, tufup sign -e 365 root <path to keystore>. I have no idea when signing should happen and what should I sign.

[dennisvang]

... When / what / how should I sign metadata files?

First let's step back a little and address "Why" you should sign metadata files

Why?

Basically, tufup downloads files from "the internet" to the computer that your app is running on, and then "installs" them.

As with anything downloaded from "the internet", there is a trust issue here: You should only run/install/use downloaded files if you trust the source.

Now, the signed metadata files are a means to establish this trust: The metadata files contain information about the "update" files that are available for download, and allows the client to verify the integrity and the authenticity of the downloaded files. The metadata files themselves are also downloaded from the internet, so they are signed to establish their authenticity.

The source of the chain of trust is the root.json file that is included in the initial distribution of your app. The initial distribution can happen via any "trusted" channel other than tufup, e.g. installation from a usb-stick or a download from your website. Initial distribution is out-of-scope for tufup.

The best tufup (or TUF in general) can do is verify that any downloaded update files come from the same source as the root.json file in your initial app distribution.

If want to know how all this works in detail, have a look at the detailed client workflow in the TUF specification. Any questions related to this should be directed to python-tuf.

When?

A metadata file must be (re-)signed whenever it is created, changed, or expired. For example:

• root.json must be signed when new "trusted" keys are added
• targets.json must be signed whenever a new target is added (i.e. a new release of your app)
• snapshot.json must be signed whenever targets.json has changed
• timestamp.json must be signed when snapshot.json has changed

What? / How?

Under the hood, the metadata files are created and signed using python-tuf, which is the reference implementation of The Update Framework (TUF) for Python.

All tufup does is provide some high-level shortcuts to make this easier. Whenever a tufup CLI command requires a <path to keystore>, that means it will update the metadata and sign it automatically.

[Eoic]

Great, thanks for the detailed explanation. It's a lot clearer now, and everything works as expected thus far.

[dennisvang]

... I use tufup Client to check for updates on start-up. It does not find any and outputs an error message, for example, "Cannot refresh metadata: root was signed by 0/1 keys". Sometimes it's "timestamp" instead of "root". I tried signing root, targets, timestamp with command, for example, tufup sign -e 365 root . I have no idea when signing should happen and what should I sign.

The " signed by 0/1 keys" error can have several causes. Have a look at 1 and 2, for example.

Did you try the tufup-example app? Please follow the steps outlined in the readme there.

You can also have a look at the example app's powershell script, which automates the steps involved in a complete update cycle.

[Eoic]

Yes, turns out it was the cache. Thanks.

[walt-jones]

@dennisvang I'm hitting what looks to be a similar issue and I cannot for the life of me figure out a way to get this working. Everything was running fine and then when I tried to shift to encrypted keys everything went wrong. Now backing out and using a fresh clone of tufup-example-master I'm still unable to get past the "Cannot refresh metadata: timestamp was signed by 0/1 keys" warning: no updates are ever found and the application won't ever update itself.

What's strangest is that the "no signature for keyid" message is not for any keyid that's actually being used... I can't figure out where that keyid is living or why it's being used instead of the keys generated in repo_init.py. Blowing away the entire temp_my_app folder with the keystore and repository and running repo_init.py to make new ones produces the same message with the same keyid. Is there some other path on disk that keys could live that I need to clear out to reset everything and get this working again?

16:37:16,521 tuf.api._payload INFO No signature for keyid 4b23b23e8cac3e9344fb362d0c023992a0bbab5a5126ab27f28178b8430d901e
16:37:16,521 tufup.client WARNING Cannot refresh metadata: timestamp was signed by 0/1 keys
16:37:16,521 myapp DEBUG No updates!
16:37:16,521 myapp INFO Done

[dennisvang]

Hi @sparkcloudstudio ,

... Blowing away the entire temp_my_app folder with the keystore and repository and ...

It sounds like the tufup client is finding some stale metadata. In that case, refreshing the repo side is not going to help. Instead, you need to refresh the client side, as discussed above, and explained in the tufup-example readme.

The client looks for metadata etc. in the UPDATE_CACHE_DIR.

Typically, on windows, that would be <localappdata>\<app-name>\update_cache, but it depends on how you configured tufup.

Here's how the client-side directories are defined in the tufup-example app:

...
DEV_DIR = MODULE_DIR.parent.parent / f'temp_{APP_NAME}'
...
if ON_WINDOWS:
    PER_USER_DATA_DIR = pathlib.Path(os.getenv('LOCALAPPDATA'))
    PER_USER_PROGRAMS_DIR = PER_USER_DATA_DIR / 'Programs'
    ... 
elif ON_MAC:
    PER_USER_DATA_DIR = pathlib.Path.home() / 'Library'
    PER_USER_PROGRAMS_DIR = pathlib.Path.home() / 'Applications'
    ...

...
PROGRAMS_DIR = PER_USER_PROGRAMS_DIR if FROZEN else DEV_DIR
DATA_DIR = PER_USER_DATA_DIR if FROZEN else DEV_DIR

INSTALL_DIR = PROGRAMS_DIR / APP_NAME
UPDATE_CACHE_DIR = DATA_DIR / APP_NAME / 'update_cache'
METADATA_DIR = UPDATE_CACHE_DIR / 'metadata'
TARGET_DIR = UPDATE_CACHE_DIR / 'targets'
...

[walt-jones]

It sounds like the tufup client is finding some stale metadata. In that case, refreshing the repo side is not going to help. Instead, you need to refresh the client side, as discussed above, and explained in the tufup-example readme.

The client looks for metadata etc. in the UPDATE_CACHE_DIR.

Thanks, @dennisvang ! We had a few different UPDATE_CACHE_DIR paths between different users and configs which made it difficult to track down. Ultimately we added some additional debug logging that included the metadata path whenever it was being read and that quickly allowed us to find all of the folders we needed to kill. Everything is now working as it should.

Looks like any time we change the keys we need to kill every client cache to avoid having it try and use the old keys. Shouldn't be an issue now that we know we need to do it.

[dennisvang]

Hi @sparkcloudstudio, thanks for the update. I'm happy to hear everything is working properly now. :-)

Looks like any time we change the keys we need to kill every client cache to avoid having it try and use the old keys. ...

The list of valid keys is maintained in the root metadata, i.e. root.json.

Whenever you make a change to any keys, the root metadata file needs to be updated accordingly. A change in keys will result in a new version of the root metadata, e.g. 1.root.json. All versions must remain available in the repository, so the client can trace them back to the version that it was originally deployed with.

This sounds complicated, but it is all handled by special key-management mechanisms provided by tufup (and the underlying python-tuf).

During initial development it may be easier just to start from scratch each time, but once you deploy your updates, that is no longer possible/desirable. So, it is important to use the key management mechanisms.

The command line interface provides a keys command, see tufup keys -h:

tufup 0.9.0
usage: tufup keys [-h] [-d] [-c] [-e] new_key_name {add,replace} ...

positional arguments:
  new_key_name     Name of new private key (public key gets .pub suffix).
  {add,replace}    Optional commands to add or replace keys.

options:
  -h, --help       show this help message and exit
  -d, --debug
  -c, --create     Create a new key pair (private & public).
  -e, --encrypted  New private key is (to be) encrypted.

In addition, the repo workflow example shows how to replace a key in a python script:

tufup/examples/repo/repo_workflow_example.py

Lines 193 to 204 in 8dad64b

	# Rotate root key (new key is not encrypted, for convenience only)
	new_private_key_path = OFFLINE_DIR_2 / 'root_three'
	repo = Repository.from_config()
	new_public_key_path = repo.keys.create_key_pair(
	private_key_path=new_private_key_path, encrypted=False
	)
	repo.replace_key(
	old_key_name='root_two',
	new_public_key_path=new_public_key_path,
	new_private_key_encrypted=False,
	)
	repo.publish_changes(private_key_dirs=[OFFLINE_DIR_1, OFFLINE_DIR_2, ONLINE_DIR])

Both approaches use the same underlying mechanism.

[walt-jones]

Thanks, @dennisvang - this is very helpful. The key takeaway here is that we should never remove old keys when new ones are added.... that's where we went wrong on this originally.

Related question: can we use tufup with pyinstaller building in onefile mode?

[dennisvang]

Related question: can we use tufup with pyinstaller building in onefile mode?

@sparkcloudstudio Yes, that should be possible, although I've never tried.

Basically, tufup just downloads files and moves them to the specified locations. It does not know what kind of files these are. It should work with any kind of application bundle, as long as tufup is configured properly.