Keeping ports up-to-date

[v1gnesh]

Hi,

I propose moving all the ZOPEN_GIT_URL and ZOPEN_TARBALL_URL to the meta repository for the following reasons:

1. If we are to use dependabot or some such automation, then it'll have to be setup for each of the port repositories (currently there are ~100 of them).
   I don't know if dependabot and such can be setup org-wide, but assuming it's a per-repository setup, this will be a hassle.

2. By moving these URLs to the meta repository, we can use a scheduled script to:

• poll the upstream sources (only a handful of them - GitHub, GNU's source control, etc.)
• submit PRs to update the _URL line(s) whenever there's a new release in one of our tracked upstreams
• keep this automation central, allowing a bot to just do stuff on the meta repo

Below commands give a quick listing of the URLs. rg refers to ripgrep.

clone all repos under ZOSOpenTools

curl "https://api.github.com/orgs/ZOSOpenTools/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone

cd to the 'parent' directory and run this

rg 'ZOPEN_GIT_URL="|ZOPEN_TARBALL_URL="' */buildenv | rg =

EDIT: Adding another neat command (to show just the URLs from buildenv files) for reference:

rg 'ZOPEN_GIT_URL="|ZOPEN_TARBALL_URL="' */buildenv | cut -d'"' -f 2

[IgorTodorovskiIBM]

Your use case for dependabot sounds really interesting.
How would a configuration for updating the ZOPEN_TARBALL_URL/ZOPEN_GIT_URL look like?
We could also have a script that automatically goes in and generates a dependabot configuration file in each repo. Also, we can update zopen generate to automatically generate it for any future projects.

[v1gnesh]

Releases from GitHub projects can be tracked with an RSS / Atom feed.
When I checked yesterday, Atom feeds were available for 'news' of some GNU projects.
So depending on where our CI is running (LoZ or zOS?), we can write something to parse RSS feed output and look for release info.
Then update the v.r.m in the URLs.

Alternatively, with rsync or rclone, we can set it up to do filtered (to most recent content) 'sync' from multiple targets (FTP or HTTPS URLs). Then look for version string, etc.

Wondering if having links centrally in meta is better as we can respond to imaginary red alert type situation (where many projects need to rebuild to address an upstream vulnerability) faster. And because meta/boot is getting cloned into each zopen setup anyway.
I do understand keeping each project self-contained, but really, they all depend on zopen.
So just thinking how much we can push up into zopen...

[v1gnesh]

To pick up latest release from GitHub, it seems this will work -

export VERSION=$(curl -s https://api.github.com/repos/ZOSOpenTools/yqport/releases/latest | jq -r .tag_name)

[v1gnesh]

Here's something that may be useful in zopen-install.
To avoid API use altogether, where possible.

https://github.com/mitsuhiko/rye/blob/main/scripts/install.sh

[v1gnesh]

Bump

I think this is now sorted.

Here are ~100 patches (covering all repos) where I've introduced bump.
patchset01.tar.gz

Background on bump:
zopencommunity/gitport#74
(Note that the above PR needs to be pulled in before gitport's bump patch is applied.)

Setup:
The following directory structure is needed.

├── zopen
│   ├── autoconfport
│   ├── automakeport
│   ├── ...
│   ├── zbump.sh
│   ├── zpull.sh
│   ├── zpush.sh
│   ├── zsshr.sh

Script name	Description
zpull.sh	Get all repos in ZOSOpenTools org, and either git clone them or git pull to update them
zsshr.sh	For each repo, change the remote origin URL to SSH instead of HTTPS, allowing usage of SSH auth when pushing
zbump.sh	Runs bump check, bump diff, and bump update. bump update can be split out; to run after review
zpush.sh	After bump update is run, check if the buildenv file is updated, commit it & create a patch from that latest commit

Right now, instead of the above purpose, I've used zpush.sh to commit & create patches for the introduction of bump.
Here's the output from zpush.sh for reference - zpush-log.txt
If necessary, the above can be put into meta/bin directory next to zopen-versions, after whatever modifications are necessary.

Once this is downloaded and setup (directory structure) similarly, bump update can be run (through zbump and zpush).
It'll create yet another bunch of patches, for repos where it has found a newer version.
They all will need to be pushed up to their respective repos through a service account on the CI system.

Future work:
Hook up GitHub Actions to existing CI somehow...
This might help, but could look around further.
It'll be a hassle setting up GHA on all repos for sure.

[v1gnesh]

@IgorTodorovskiIBM, I've got a solution now for sudo too.
Will do a normal PR for it once the existing patches are pulled in.
I think reviewing & merging 5-10 a day is doable :)

sudo-version /SUDO_VERSION="(.*)"/ https://www.sudo.ws/getting/source/|re:/sudo-([\d.]+p?\d?).tar.gz"/$1/

[IgorTodorovskiIBM]

That sounds good to me! We should def have a meeting to discuss how we can integrate this into our CI pipeline (or consider GH actions)