Have komodo build a release from root packages

[jondequinor]

Rationale:
For any given root package (like ert-2.4.0), its dependencies are not deducible from a repository alone. Currently, a repository only describes dependencies as sets of packages, and the actual packages in that set of dependencies are defined in a release.

This is unfortunate for a bunch of reasons, but mostly social. For example, if ert-2.4.0 is to depend on libres-4.1.1, a bump from libres-4.1.0, a release manager can and will not bump ert, but simply bump libres. I.e. ert-2.4.0 depends not only on libres-4.1.0 but also libres-4.1.1. This creates ambiguity; what is the real libres dependency of ert-2.4.0? This varies from release to release, such that to reason about ert you also need to reason about any given release in which it reside.

Technically, it is cumbersome, because without releases, the dependency graph is woefully incomplete.

Proposal:
Ideally, a release is a set of root packages (ert-2.4.9, everest-4.5.6), and all the root dependencies are deduced from a repository.

For the case above, in order to have ert-2.4.0 depend on a newer libres, you would need to edit the repository directly, which will be severely frowned upon. If the released packages in the repository are considered immutable, and you can only create releases of root packages, you're forced to bump ert as well.

For this to work, we need to

• make every dependency a package, meaning all dependencies in komodo will be in the format - libres: 2.4.0 or - libres: 2.4
• introduce matrix semantics into the repository format. So dependencies can have the format:

ert:
  2.4.0:
  depends:
      rhel7:
        py36: 
          - pyqt5: 5.13.0
        py27:
          - pyqt5: 5.13.0-src

• allow releases to be a set of root packages, and nothing else. A release with only ert will thusly look like this:

ert: 2.4.0
# ert's dependencies are deduced

#70 would be easier to solve, since we get a concept of a root package.

[markusdregi]

Thanks for the nice write up 👏 Probably a stupid question, but if you then need to deploy a bug fix in libres; how would you go about it?

[pinkwah]

Alternative proposal:

Keep dependency list as is:

ert:
  2.4.0:
    depends: [libres]

However, package that has versions which have incompatible versions are marked as such:

numpy:
  1.18.4:
    only: [py36]
    # ...
  1.16.4
    only: [py27]
    except: [rhel5]
    # ...

Then komodo goes over every dependency of ert, and selects the latest compatible version (top to bottom, komodo can safely ignore parsing versions) and spits out a release file (aka lock file).

To generate a new release file from scratch, we can use the "public" packages as described in package_status.yml together with repository.yml. An example tool could be as follows:

$ komodo new ~/komodo-releases > ~/komodo-releases/releases/matrices/2020.10.0.yml

# Debugging komodo problems, generate release with only ert version 2.0.0 and all of its dependencies
$ komodo new ~/komodo-releases ert==2.0.0

Dealing with semver breakage:

We just don't. If libres 10.0 is released and is incompatible with current ert, then don't add it to repository.yml until it is. When ert is compatible, we add both and then everything is fine and dandy.

All other problems can be found during testing.

Backporting fixes:
If 2020.10.0 is out, but we need to bump libres in 2020.9.0, then we just do it as we do today: copy the 2020.9.0 release and create 2020.9.1 with the libres bumped.

[jondequinor]

Thanks for the nice write up clap Probably a stupid question, but if you then need to deploy a bug fix in libres; how would you go about it?

You would have to create a new ERT release as well.

[jondequinor]

I back @dotfloat's proposal. If we have a way to share our algorithm for creating the dependency tree, tooling can easily be built on top of it.