Cache all root metadata versions

[jku]

I suppose this is ready to be a draft -- at least I won't lose the work this way.

Problem

Currently python-tuf clients typically initialize once from the "bootstrap" root (the root metadata shipped with the client software), subsequent startups will use the cached root as starting point. This is mostly good but there is a downside: if the bootstrap root is more secure than the cache, then we lose some security (as cache could get poisoned between the first cache initialization and the actual use). This situation can arise if the client software (and bootstrap root) is shipped in a Operating System update, container image or e.g. a Debian package: in this case the client cache is writable by user but the bootstrap root may not be.

Solution

• Cache all root versions
• Always initialize the client starting with the secure bootstrap root
• When loading subsequent root versions, load from local cache and from remote repository

This should give us best of both worlds:

• client always initializes from most securely stored root
• client does not download any more data than now
• client still uses local cache for roots so repository cannot serve obsolete data

The only downside is that client will now verify every root version on startup: I think this perfectly acceptable as client can and should periodically update the bootstrap root as well -- the added computation is not an issue in practice.

Details

The ngclient cache directory looked like this before:

cachedir/
    root.json
    timestamp.json
    snapshot.json
    targets.json
    somedelegatedrole.json

Now it looks like this:

cachedir/
    root.json        #  This is a symlink to "root_history/3.root.json"
    timestamp.json
    snapshot.json
    targets.json
    somedelegatedrole.json
    root_history/
        1.root.json
        2.root.json
        3.root.json

• non-versioned root.json is preserved as symlink so that older python-tuf versions can coexist (and so it's easier to check current state for humans and tests)
• root_history directory is used to make name clashes impossible (imagine a delegated role named "1.root") : as long as the client is not susceptible to path traversal issues, this should be safe
• unused test data was removed in this PR, some tests were cleaned up to make them work with new cache structure
• tests have been added to verify that roots are cached, cached roots are used and that poisoned cache is handled correctly (remote metadata is used if cache is not valid)

There is a cleanup PR that should be handled before this one: #2768

[jku]

I think I'll refactor the large test cleanups to a different PR so this doesn't look so big (it isn't)

[kairoaraujo]

I think I'll refactor the large test cleanups to a different PR so this doesn't look so big (it isn't)

I'm ok with reviewing it if you want to add to this PR (separate commit).

[jku]

not planning to add anything: I just meant the test file removals that are already in the PR: "+268 −1,253" looks like a big change but the actual PR is really manageable

[jku]

OK:
there is now #2768 that should be handled before this one: this changes this PR into a fairly small one: 235 insertions(+), 89 deletions(-)