What tool should be used to checkout NorESM externals?

[gold2718]

The NorESM 2.5 alpha08 tags introduced git fleximod as a replacement for manage_externals. Both tools were developed at NCAR and both tools aim to manage the correct checkout of NorESM externals and sub-externals based on a documented set of external repositories and tags. Experience with git fleximod has led to this discussion on the properties of a desired solution for the NorESM community and developers. This introduction has two sections:

• Should the NorESM external checkout tool use git submodules
• What features should the NorESM external checkout tool have
  • Required features
  • Desired features
  • Questions

Should the NorESM external checkout tool use git submodules

• git fleximod checks out externals as submodules. Because the tool uses the .gitmodules file to document the desired tags, it also requires that the submodules exist in the commit.
• manage_externals does not use submodules. Each external is checked out as a separate clone.

What features should the NorESM external checkout tool have?

• Required features

  • The tool reads a configuration file (e.g., External.cfg, .gitmodules) and checks out the correct reference of each external
  • The tool is recursive, it has a way to make sure that sub-externals are properly checked out
  • If a new commit is checked out, the tool can switch the checked-out reference for each external
  • The tool will not delete any local (i.e, not committed to a repository) work without a --force option
  • The tool can report the status of the external checkouts in a few easy to read and understand formats
    • If the tool uses git submodules, it must include submodule synchronization information in status reports
  • git submodules in a checked out external will be updated correctly
  • The tool is stable and will work on a broad range of operating systems and versions
  • The tool must support sparse checkouts and sparse cloning
  • The tool must have well defined behavior and error conditions
    • Any behavior not explicitly defined is an error?
  • The configuration must allow for different externals to be checked out if a repository is a sub external or a top-level repository
    • Configuration files (<name>.cfg) manage this by using one file for top-level checkouts and one for sub-externals
    • If submodules are used, there must be some way to control which submodules are checked out for a sub-external.
      • One possibility is an option to disallow duplicate checkouts.

• Desired features

  • The tool can be invoked on a limited set of externals
  • The tool has an exclusion mechanism that can be invoked on a top-level external or a sub-external
  • The tool should provide a way to update the configuration file based on the currently checked out externals
    • This update may or may not commit any changes, possibly determined by a configuration flag
    • If the tool uses git submodules, it should also update the top level repository to match
  • The tool should support shallow clones
  • If a branch is specified instead of a tag or hash, it should be checked out as a branch, not a detached HEAD.

• Questions

  • Should the tool be atomic? That is, if any external update fails, should the tool restore the previous settings?

[oyvindseland]

I must admit that for my part I do not have very strong preferences, so hard to unless I really go through the details. I have a couple of comments though.
-It should write out a log of what your command actually did to your settings / existing checkouts.

Should the tool be atomic? That is, if any external update fails, should the tool restore the previous settings?
I guess that this will be hard, but would a dry run be possible / easier to implement than a full restore, ideally with proper warnings for fails?

Any behavior not explicitly defined is an error?
Depends on how often this is expected to occur. At least I think an error should be defined if the behavior risk doing something irreversible to the system (the --force in example). If it is not critical I think that ideally the behavior should be defined as an error if the error message comes with a suggestion for how the issue can be resolved, syntax, bounds etc.

[gold2718]

I am starting to wonder if we want to stick with submodules at the top level of NorESM. With manage_externals, you could have two different Externals.cfg files which could be used to checkout different versions of the model. This is not possible with either git fleximod or with plain git submodules. Does anyone see a use for this potential feature?

[TomasTorsvik]

@gold2718 - I think it's a good point that we might want different Externals.cfg files. I find the git submodules/fleximod to be more difficult to work with when switching between model versions (usually I end up cloning everything again from gitHub sources). Maybe it would be easier if we use submodules for dependencies within components.