How to correctly list check names for merge protection rules

[bartgol]

I have been struggling trying to add the correct check names to my protection rules. I am hoping someone can pin point my mistake.

I have a few jobs, which can run or be skipped (e.g., if a certain label is found, I skip the job), so I want my check to be "either they passed or they were skipped". I realized that the check name was not "/", so to be safe I used PyGithub to retrieve the exact names. This is what I got:

 Check Name: gcc-cuda / dbg
 Check Name: gcc-cuda / sp
 Check Name: gcc-cuda / opt
 Check Name: gcc-openmp / fpe
 Check Name: gcc-openmp / dbg
 Check Name: gcc-openmp / sp
 Check Name: gcc-openmp / opt
 Check Name: cpu-gcc / SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.scream-mam4xx-all_mam4xx_procs
 Check Name: cpu-gcc / ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-small_kernels--scream-output-preset-5
 Check Name: cpu-gcc / ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.scream-dpxx-arm97
 Check Name: cpu-gcc / ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.scream-output-preset-2
 Check Name: cpu-gcc

My current .mergify.yml file has this

    success_conditions:
      - "#approved-reviews-by >= 1"            # At least 1 approval
      - "#changes-requested-reviews-by == 0"   # No reviewer asked for changes
      - or:
        - check-success="gcc-openmp / .*"
        - check-skipped="gcc-openmp / .*"
      - or:
        - check-success="gcc-cuda / .*"
        - check-skipped="gcc-cuda / .*"
      - or:
        - check-success="cpu-gcc / .*"
        - check-skipped="cpu-gcc / .*"
      - or:
        - check-success=cpu-gcc
        - check-skipped=cpu-gcc

While the last one is picked up (corresponding to the last check in the list above), the othres are not. I tried to modify the syntax, since I figured the spaces may be a problem. E.g., for gcc-openmp / XYZ I tried the following

• check-success=gcc-openmp/*
• check-success=gcc-openmp / *
• "check-success=gcc-openmp/*"
• "check-success=gcc-openmp / *"

None of them worked. What am I doing wrong? How can I capture a bunch of checks (whose name contains a space) in one "check-success" clause?

[jd]

You're using = operator but .* sounds like you want a regexp.
You should write `check-success ~= "^cpu-gcc / .*".

Also, be aware that this will be true as soon as one job matching this regex is true.
I'd suspect you want to list the expected jobs.

[bartgol]

That's right. I want ALL the jobs matching the regex to either pass or be skipped. My AI bot told me that using a wildcard would mean that ALL checks matching it would have to all pass or all be skipped (which is what I want in my case). But given what you said, I guess I'm forced to spell out the name of each test?

[jd]

There's no such thing as "all checks" because checks in GitHub are dynamic. There's no way to know which checks are or are not expected on a particular PR.
That's why you must list them all; otherwise, a PR could be merged even if a single check matches.

[bartgol]

That makes sense. Thanks!

[bartgol]

@jd I tried the following:

- or:
  - and:
    - check-success="gcc-openmp / dbg"
    - check-success="gcc-openmp / opt"
  - check-skipped~="gcc-openmp / .*"

but it doesn't work. The problem is that these dbg/opt/... (there are more) are jobs created by a matrix strategy. I use the matrix entry to create the job name, like

  name: my-job / ${{ matrix.build_type }}

and if the job is skipped, there is only ONE check, whose name is the literal string my-job / ${{ matrix.build_type }}. I tried to use that, but mergify seems to interpret ${{...}} as a variable to expand, which is why I tried to go back to your suggestion about regex, and using ~=. But I'm clearly doing something wrong. Using PyGithub, I can see the check name " my-job / ${{ matrix.build_type }}", but I don't know how to make mergify pick it up as a skipped check...

[jd]

You can use this:
- check-skipped = {% raw %}my-job / ${{ matrix.build_type }}{% endraw %}
This will prevent any interpretation done on the job name since it contains {{ which matches the Jinja2 template format that Mergify uses. This avoids using a regex — though I don't see right now why it does not work.

In any case, if you can share a link to a PR that does not work as you want, feel free to share here or privately so we can help debugging.

[bartgol]

Thank you very much for you quick help. I will try it right away.

[bartgol]

Since you were puzzled by the fact that it wasn't working, here's a screenshot of the checks tab with the regex syntax. In case you can see some other issue.

The checks names are cut off, but the cut off part is just "XXXX }}", with XXXX some sequence of chars

[bartgol]

Confirming that the "raw" string trick did it: