Fixes #37139 - Block content view publish during repository publication tasks

[sjha4]

What are the changes introduced in this pull request?

Blocks content view publish task while certain repo actions (Sync/upload/remove content) are in progress. Similarly block these repo actions when corresponding CV publish is in progress.

Considerations taken when implementing this change?

There are multiple places in CV publish orchestration where we fetch repository contents. If repository contents are changed during a publication, different steps in the publish workflow will receive different versions of the repository with possibly different contents. This would lead to improper cv publications.

What are the testing steps for this pull request?

Have a few repos and cvs with those repos.
Try a combination of actions like sync/upload/remove content/generate metadata and associated CV publish to make sure the 2-way blocking works as expected.

[sjha4]

[test katello]

[wbclark]

Attempted to publish a CV while one if its repos was syncing:

500
Pending tasks detected in repositories of this content view. Please wait for the tasks: - http://centos8-stream-wclark-katello-devel.example.com/foreman_tasks/tasks/7590b00d-5532-49bf-9cbc-2377d1bca5de before publishing.

The CV publish task immediately went to stopped/error. The repository sync task remained running/pending.

[wbclark]

Summarizing an offline chat we had about this:

1. Should this blocking behavior be controlled by a setting, allowing users with hundreds of repos, automated sync plans, etc to opt out of the blocking behavior to avoid these collisions? Is the current pre-blocking behavior that the CV can be published with the exact repo content that existed prior to the sync, or is there any data inconsistency possible due to the repo contents changing during publishing?

2. Is there a way that we can block these tasks, still clearly informing the user of what is happening when they try to initiate it, without having a "scary" red stopped/error task showing up on the monitor tasks page thereafter? E.g. could we check for blockers and surface them before the task even gets created, or else have the task go to stopped/warning instead (without changing the entire rescue_strategy for other modes of failure, and still surfacing a UI notification explaining what is happening?) @adamruzicka do you have any ideas about this?

[sjha4]

Should this blocking behavior be controlled by a setting, allowing users with hundreds of repos, automated sync plans, etc to opt out of the blocking behavior to avoid these collisions? Is the current pre-blocking behavior that the CV can be published with the exact repo content that existed prior to the sync, or is there any data inconsistency possible due to the repo contents changing during publishing?

After looking at the CV publish code flow, I am leaning towards blocking it for everyone to prevent any changes between repo contents when the publish starts to when it finishes. Cause many actions in the code path independently looks up repository and contents during the flow and changing repository between these sub actions would lead to unexpected results in the published version.

E.g. could we check for blockers and surface them before the task even gets created.

We could potentially move the validations to the API layer and block creation of task. However syncs and other blocked repo actions are started from multiple places outside of API. ex: sync plans which directly trigger sync action. Having the blocking at action layer would handle all cases including foreman-rake invocations or any rake tasks. I don't think we need exclusive locks on repo during sync or any other actions however and a custom validation to only block publish and nothing else would suffice for this.

[sjha4]

[test katello]

[wbclark]

LGTM

We can change one commit from "Fixes" to "Refs" or squash on merge

[wbclark]

do you anticipate that we may use this in other areas in the future?

[sjha4]

Not necessarily..However, this is the list of tasks with potential to create a new publication. So if we need to block anything on itthe future, a repo.check_ready_to_act! should work for that use-case.