Discuss Server Repository Structure

[DominicOram]

As a developer I would like to minimise the amount of time I have to spend switching branches, whilst still making sure things are tidy. We should have a discussion as to whether the current repository structure does this.

Acceptance Criteria

• A discussion is held about how we could improve our repository structure
• The agreed new structure is documented and a ticket created to implement

[FreddieAkeroyd]

Some things to think about

Do we have too many repositories and are we using submodules right.

For third party epics modules I think separate repositories work well, they can be swapped/upgraded/versioned on separate branches, they do not change very often, and we may want to replace then with a fork rather than import a vendor branch at some point.

When we add a new stream device ioc, we end up with a new support repository, then branches in ioc, device emulator, ioc test, css screen. Do we want to combine these more e.g, should the emulator, test, css, ioc etc all be in the same (new) repository? Or maybe share a repository for some of these elements

Potential alternatives to submodules: git subtree, google repo https://android.googlesource.com/tools/repo

[DominicOram]

As part of this discussion we also need to work out the structure of IocTestFramework (#6093)

[DominicOram]

Will affect the way the IOC generator works (e.g. like #3431)

[FreddieAkeroyd]

Some repository management tools:

• google repo https://gerrit.googlesource.com/git-repo but windows support is best effort https://gerrit.googlesource.com/git-repo/+/HEAD/docs/windows.md
• chromium gclient https://dev.chromium.org/developers/how-tos/depottools and https://pypi.org/project/gclient/
• jiri https://fuchsia.googlesource.com/jiri/
• git subrepo https://github.com/ingydotnet/git-subrepo
• git subtrees https://www.atlassian.com/git/tutorials/git-subtree https://tsh.io/blog/git-subtree-or-how-i-stopped-worrying-and-learned-to-love-external-dependencies/

https://www.edureka.co/blog/git-submodules-versus-googles-repo-tool

[DominicOram]

Can we rename master to main at the same time?

[John-Holt-Tessella]

• rename directory called master to subrepo or something similar

[davidkeymer]

Could this be an opportunity to rename our account(?) from the now obsolete "ISIS Computing Group" to the current "ISIS Experiment Controls"? I realise this may not be possible and even if it is, may cause more trouble than it's worth!

[DominicOram]

Meeting booked for 15:00 on the 24th May

[FreddieAkeroyd]

Some thoughts

Third Party repositories

Current workflow

Many of our third party repositories, such as EPICS base and other support modules like asyn, are currently done in a “vendor branch” style. Here an update workflow is:

• Unpack tar file and add new external version to a “vendor” branch in our repository and tag
• Create an “update_to_XXX” branch based on our master
• Merge the update branch

Advantages: no branch naming conflict or imposition from the original (upstream) repository.
Disadvantages: need to obtain and merge a new tar/zip file of files; no link to original repository, so no history, and not easy to create PR to send back

Fork workflow

If the vendor repository is on GitHub, then it could be forked

Advantages: full history, no need to import tar file of new release – can merge new release tag tag.
Disadvantages: creating a PR will default to the upstream repository, so need to be careful. Also need to consider potential branch name conflicts. Should your local origin/main reflect upsteam/main or diverge?

Clone workflow

A bit like fork, you clone original repository and push to one in our organisation. There is no direct upstream link like with a GitHub fork, so PRs will be against local repository. You get commit history, but will only see branches and tags that you decide depending on how you do the original clone and push.

Advantages: more control, less confusion with PRs
Disadvantages: relationship with the original repository is less clear, no so easy to create a PR back to original repo.

Reducing number of repositories

Adding a new IOC results in many repositories: support module for db/protocol, branch in ioc repository, branch in emulators, branch in ioc system tests, branch in ibex_gui for css screen. Many of these are related, can we reduce?

For a stream device ioc you could put db and proto into EPICS-ioc, but still the question of other modules. Would it be useful to be able to put more into the “support” module? For example a “put everything in support” approach would need to consider:

• Existing EPICS-ioc structure: this could be kept, but assembled dynamically at build time. So support modules would have an “iocs” subdirectory, and at build time a search would be made to locate “iocs” to assemble something like the existing EPICS-ioc structure. Alternatively an iocs structure could be created with redirects to appropriate support module locations.
• Ibex GUI CSS resources. As part of build, these would need to be assembled somehow. Maybe an epics build publishes all the opis to a shared folder that a gui build can use? Or could the opis be assembled into an “opis” folder in the epics tree, and a gui reference these at runtime via share or web? This would mean a GUI could look at instruments with different OPIs for their device screens as the appropriate OPI would be loaded from the instrument.
• It may or may not be necessary to assemble emulator/ioc test files during build. Of maybe a “make runioctests” at the top level recursively runs them as it finds them?

[rerpha]

This is a wild and probably very stupid suggestion:

Could ioc become a monorepo with the support modules built into the root of the directory (not as submodules)? this would reduce the number of repositories and submodules, meaning you only have to create 1 branch to create or modify an IOC.

advantages:

• initial cloning is easier, you only have to pull master and you get everything up to date at once
• only have to create and maintain one branch if you are creating/updating an IOC
• We'll only be using up one private repo for ioc versus several for the individual support modules - though this means ioc would have to become private if there are support modules which are private
• Releases become a case of checking out one branch rather than checking out every repo inside of EPICS

disadvantages:

• initial clone will be much slower because you are going to be pulling a huge amount of data. Once you're set up it might be quicker? although maybe not - git is quite slow to do anything in epics-top currently.
• merge conflicts may be nasty and things may get reverted by accident in the support module
• you lose per-module versioning which would make things like reverting stuff back on instruments very difficult and messy
• Not very easy to just checkout one part of support if you needed to quickly clone it, make a change and push it

I understand the git performance may be the biggest limitation here, but i'm not sure how bad it'd be without testing it. Maybe we could use git LFS to speed things up? maybe we should do this anyway for binaries and so on? I don't know enough about it to say.

Alternatively, could support become the monorepo and we keep ioc separate? that way you only create two branches at most rather than 3+ in some cases?

[ChrisM-S]

Probably even more radical, but would it make sense to do both as monorepos? The key here would be to use a file system link link to the support modules from the IOC repo. On compile, dependencies/includes would be picked up from the support module at compile time and built libraries from the same at link time. The .exe would refer to exactly the same paths because I assume the support module structure would still be there on disk at runtime in a normal deployment (and any version information would still be included in the build). The support modules would also be simple in that they would be a complete item.

1. Exporting a complete IOC to somewhere out of our system would then be just a case of copying the visible IOC tree to a separate place (the support modules would be copied along automatically).
2. Both would be modular when built(although the support modules would essentially be a modular library?).
3. Maybe good/maybe bad, built IOCs could potentially be run with different version of support modules (i.e. do copy as above into deployment area if required). Building an earlier version of the support modules would also be easier as there would be no dependencies, just ensuring that the right versions were linked.

Probably a hole in here somewhere, but being able to modularize IOCs in (particular) to nothing more than agreement on basic EPICS/IBEX conventions might allow a single IOC to be exported on it's own - perhaps on a visiting users laptop - with no IBEX installation.

[KathrynBaker]

As we are postponing the discussion, I'll note here my one thought based on the above two comments, whilst it might not be as clean in some ways we could have two separate trees for public/private repos, whilst not as clean as just a single tree it would allow greater sharing where we can do so and maintain the privacy where that is required.

[DominicOram]

In the meeting we decided that we would like to reduce the number of submodules that need to be checked out for each IOC and also allow an IOC to be more easily shared with other facilities. We would also like to generally do some tidying up of our server-side file structure. To this end we have created the following tickets (in rough priority order):

• Modify the IocTestFramework to be able to run the tests on a self contained "support" folder that holds the tests and emulator. This should be run by running "make ioctests", and a recursive "make ioctests" from the top will run them all (Move one emulator and IOC test to a support directory #6555)
• Move tests and emulator into support for every IOC (Move emulator and IOC tests into support module #6556)
• Ticket to evaluate ISIS folder, general folder structure, do we need a dev area for dbunittests etc. (Evaluate server file structure #6557)
• Discuss how we do forking etc. of third-parties (Discuss our process for updating third party moduels #6558)
• Rename master into main (Rename master to main #6559)
• Move a st.cmd fragment into support that will hold the connection information (baud rate etc.) and the db load (Add a cmd fragment support directory #6561)
• Move OPI onto server (needs a bit more thought probably) (Discussion: Move OPIs to server #6562)
• Another discussion for whether we like submodules (Discussion: Move away from submodules #6563)
• Update ISIS Computing Group to Experiment Controls - check what this will break (Update ISISComputingGroup to ISISExperimentControls #6564)

[rerpha]

Was in the discussion and the relevant tickets have been created.