Use the new version for the McM REST client

[ggonzr]

Related to: #23, #1138

Use the new version for the McM REST client

1. Remove all the instructions that load this module from AFS. Defer to the user the responsibility of preparing the execution environment to include this package.
2. Raise a detailed exception explaining how to install the new version if the module is not available.
3. Update instructions that used mangled methods available in the McM class to avoid DeprecationWarning messages.
4. Delete some sections that look like dead code

[ggonzr]

Hi @efeyazgan. These changes are related to using the new version for the McM REST client and avoiding importing modules from AFS. I saw there are some CI jobs to check changes for the script. However, they do not check the contribution provided in the PR (the commits differ - PR: a944f5b41fbf1bd9631855c68bbed7ba195ded30 vs used by the CI: 38b30bdba0758e27e99152958873b073312a0b17). Could you test this PR and let me know if you agree to merge this contribution?

Thanks,
Geovanny

[efeyazgan]

Hi @ggonzr
When I test this locally, this is what I get:

./run_req.sh EXO-Run3Summer22wmLHEGS-01444
---> 1 request will be checked:
 
Current date and time: 2024-08-26 16:30:28
Prepid(s):
EXO-Run3Summer22wmLHEGS-01444
Traceback (most recent call last):
  File "/afs/cern.ch/work/e/efe/GENprod_gio/genproductions/bin/utils/request_fragment_check.py", line 497, in <module>
    res = get_request(prepid[num])
  File "/afs/cern.ch/work/e/efe/GENprod_gio/genproductions/bin/utils/request_fragment_check.py", line 72, in get_request
    result = mcm._get('public/restapi/requests/get/%s' % (prepid))
NameError: name 'mcm' is not defined
[efe@lxplus919 utils]$ vi run_req.sh 
[efe@lxplus919 utils]$ ./run_req.sh EXO-Run3Summer22wmLHEGS-01444
---> 1 request will be checked:
 
Traceback (most recent call last):
  File "/afs/cern.ch/work/e/efe/GENprod_gio/genproductions/bin/utils/request_fragment_check.py", line 57, in <module>
    from rest import McM
ModuleNotFoundError: No module named 'rest'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/afs/cern.ch/work/e/efe/GENprod_gio/genproductions/bin/utils/request_fragment_check.py", line 65, in <module>
    raise ModuleNotFoundError(import_msg) from e
ModuleNotFoundError: Unable to import the McM module. Install this module in your execution environment by following the instructions available at: https://github.com/cms-PdmV/mcm_scripts

This happens because of

genproductions/bin/utils/request_fragment_check.py

Line 55 in a944f5b

if args.local is False:

when running locally.

Is this new method to be used in the same way in lxplus (locally) and elsewhere? Then, I can drop that if.

However, even when I drop that statement I get the same error.

Before running this command, I do

python3.9 -m venv venv && source ./venv/bin/activate

Am I missing anything?

[ggonzr]

Yes, you need to install the package by executing this pip instruction. By the way, is this script meant to be used only by the McM application, or do other users/applications use it?

[efeyazgan]

OK, I'll try and let you know.

This is being used both by McM and many users locally (in lxplus).

[efeyazgan]

OK, after installing the package, it worked. I think it is OK to merge.

[bbilin]

Dear all,

One point before we merge, this script is run on condor (unless you changed this recently), so installation should be also implemented in the script itself, no?

Thanks
B.

[efeyazgan]

Is there a script running the request checking script on condor? If so, that should be updated.

[bbilin]

I think it is the get test that is executed on condor. (IIRC, current PdmV and Geovanny can correct me if wrong). So indeed it can be added on the line before the script is run.

Thanks

[ggonzr]

Indeed, this script is embedded to run McM validation jobs in HTCondor and prepare other kinds of scripts that the application provides. But as described before, #1138 takes care of this.

The only remaining thing would be to send an announcement to other users about this change, which is a breaking change in case they run the script directly in their workflows as they have to also prepare an environment that installs the package. Do you know if this situation happens often and who could be affected by this?

Perhaps, instead of raising an error, we could try to temporarily import the package available in AFS and raise a DeprecationWarning alerting on this change giving users a window of 3 weeks to 1 month to update their environments before we cut the support on this old version. Do you agree with this approach?

[bbilin]

Hi Geovanny Thankw for clarification, then indeed i would add a warning and a deadline, then we can switch to error. Cheers B Envoyé de mon iPhone Le 30 août 2024 à 09:57, Geovanny González-Rodríguez ***@***.***> a écrit :  Indeed, this script is embedded to run McM validation jobs in HTCondor and prepare other kinds of scripts that the application provides. But as described before, #1138<cms-PdmV/cmsPdmV#1138> takes care of this. The only remaining thing would be to send an announcement to other users about this change, which is a breaking change in case they run the script directly in their workflows as they have to also prepare an environment that installs the package. Do you know if this situation happens often and who could be affected by this? Perhaps, instead of raising an error, we could try to temporarily import the package available in AFS and raise a DeprecationWarning alerting on this change giving users a window of 3 weeks to 1 month to update their environments before we cut the support on this old version. Do you agree with this approach? — Reply to this email directly, view it on GitHub<#3750 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABQ4O2VE2B7JMIN62RXHWJDZUAQXDAVCNFSM6AAAAABMZ3FYBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRQGQYDGMZZGQ>. You are receiving this because you commented.Message ID: ***@***.***>

[efeyazgan]

That seems OK (for the ones who want to run locally, I summarized the commands here: https://github.com/cms-sw/genproductions/blob/master/bin/utils/run_req_new.sh)

[ggonzr]

Hi @efeyazgan,
Please could you test this again, the last commit loads the old version from AFS if the user has not provided a proper venv with the latest version of the McM package.

[ggonzr]

I just rechecked this with the integration for McM described in #1138 and it works by providing a venv or loading the module from AFS directly. Therefore, this PR is ready and you can merge it. Please do not forget to send an announcement to your stakeholders about this change (CMS Talk perhaps). Next month I will create a PR to remove this backward compatibility so that we do not rely on old versions

[efeyazgan]

Sorry. I had missed this. I tested it and it works OK. Thanks a lot!

[efeyazgan]

@bbilin @lviliani please take a look. Thanks.

[lviliani]

Thanks! Looks good to me, I only have one question to make sure I understand.

I think MC contacts usually download the test script from McM (using the "Get test command") to run some quick validation locally on lxplus, before running the McM validation.

Can you please confirm that this will still work, provided you merge also cms-PdmV/cmsPdmV#1138?

[ggonzr]

Yes, it will work even if cms-PdmV/cmsPdmV#1138 is not merged immediately as the script will load the old version without issues if it is executed in Lxplus nodes. So it is safe to merge this