[chore] [WIP] Add script that runs go mod tidy in topological order

[jade-guiton-dd]

Context

In a repo with multiple Go modules, some of which depend on each other, and where internal dependencies are managed with replace statements, you end up with the situation where an update to the go.mod of one module can require running go mod tidy on modules that depend on it.

We have the make gotidy command available, which runs go mod tidy on every module in the repo. But this is done in alphabetical order, so that if alpha depends on bravo which depends on charlie, and charlie's go.mod is modified, bravo will get an update, but not alpha, as it gets tidied before the other two.

This leads to sometimes needing to run go mod tidy multiple times before things converge. This caused an issue in the latest release process in core, where the recently modified (link to relevant PR) check-contrib check failed because running make gotidy only once caused an error when running make generate (link to the relevant issue).

Description

This PR attempts to solve this issue by introducing a make topotidy command, which runs go mod tidy commands in topological order, ie. in an order such that dependencies always come before their dependents. This makes sure that changes to a go.mod are always fully propagated through the dependency graph.

Unfortunately, topological order doesn't quite cut it, as there are loops in the dependency graph (at the moment, between the two Datadog components, and between the three OTel-Arrow modules). It's not entirely clear if convergence is even possible in this context, but at the very least, we may need to run go mod tidy on a module twice to guarantee that all changes have had a chance to be propagated to every other module.

To solve this, I used Tarjan's algorithm, which sorts the graph topologically while isolating its strongly connected components (SCCs, parts where no topological order is possible), and applied a very naive algorithm to each SCC (which will hopefully remain few and small).

I implemented all this as a Python script in the internal/buildscripts directory. I believe this is the first Python script in this repo, so please reach out if there are concerns about this.

Link to tracking issue

Updates this issue on core. Unless we decide to replace make gotidy entirely with this script, we will need a PR on core to use make topotidy inside make check-contrib.

Testing

I locally replicated the latest release process, and checked that all "updates to go.mod needed" were eliminated when using make topotidy instead of make gotidy inside make check-contrib.

Documentation

I added a comment explaining the rationale and functioning of the script at the top of the script.

[atoulme]

Consider having this list generated in a file so we can consult it, and update it via script?

[jade-guiton-dd]

That's probably a good idea, although I'll probably need to implement yet another CI check to prevent the committed version from being out-of-date.

[atoulme]

No free lunch on that one

[jade-guiton-dd]

What about outputting the list to a .gitignored file?