Build a MSUnmerged docker image based on Almalinux

[amaltaro]

Impact of the new feature
MSUnmerged (but we might decide to extend it across WM services)

Is your feature request related to a problem? Please describe.
Yes, the gfal2-util package is only maintained in the EPEL RPM repository. Given that our base image is debian based, we cannot fetch updates made to GFAL2 stack, where we rely on third-party packaging/updates to Anaconda repositories. This is the current mechanism to get gfal2 into our MSUnmerged image:
https://github.com/dmwm/CMSKubernetes/blob/master/docker/pypi/gfal/Dockerfile

Describe the solution you'd like
We have been discussing pros/cons of switching our base image from Debian to what WLCG is adopting (e.g. Almalinux). Many concerns have been expressed with this change though, e.g.: a) image size due to inner-dependency in RPM systems; number of vulnerabilities; and maintenance in general (including the python stack).

With this issue, we would like to prototype a new MSUnmerged docker image based on Almalinux 9 and evaluate it against the 3 concerns listed above, compare it with the current image and consider anything else that might be relevant.

Note that this issue is targeting the MSUnmerged service, which is in the critical path with lack of updates to the GFAL2 stack. Nonetheless, we can also consider updating the WM services to adopt the base Almalinux 9 image, but this will come in the future.

Describe alternatives you've considered
None

Additional context
None

[amaltaro]

Initial prototype is provided in this PR: dmwm/CMSKubernetes#1452
with very interesting/positive results! I have not tested this service with production payload (e.g. loading different storage plugins), but it seems to be functional.

Current open questions are:

• we need to evaluate usage/alternatives for the rotatelogs tool
• we also need to consider what to do with the python version (stick to the currently available 3.9, or deploy our own stack on py 3.8)

[amaltaro]

This ticket so far depends on the following CMSKubernetes ticket:
dmwm/CMSKubernetes#1452

and this WMCore PR - which removes rotatelogs:
#11955

Note that the test reqmgr2ms-unmerged:2.3.2rc6-20240419 image no longer depends on rotatelogs.