feat: validate kustomize build of tasks and pipelines #1792

tisutisu · 2025-01-07T07:59:15Z

This PR adds a CI check to validate kustomize build is successful and also checks that PR authors are committing built manifests files of task/pipeline change as well with their task/pipeline changes.
Part of story: https://issues.redhat.com/browse/STONEBLD-3042

chmeliik · 2025-01-07T14:44:47Z

hack/build-manifests.sh

+    cd "$SCRIPT_DIR/.."
+    task_dirs=$(find task -maxdepth 4 \( -name 'kustomization.yaml' -o -name 'kustomization.yml' \) -exec dirname {} \;)
+    pipeline_dirs=$(find pipelines -maxdepth 3 \( -name 'kustomization.yaml' -o -name 'kustomization.yml' \) -exec dirname {} \;)
+    local out=auto-generated


Instead of prepending the auto-generated/ prefix, can we put the generated files into the original directory? e.g. task/build-image-index/0.1/build-image-index.yaml?

I think that would automatically make the build-and-push.sh script notice that the task yaml already exists and it's not necessary to run kustomize again

In that case, we have to also update the build-and-push.sh script and don't run the oc kustomize cmds again during build-and-push.

looks like we don't need to modify the build-and-push script, if we commit the tasks and pipelines to the same directory, made the changes now.

yftacherzog · 2025-01-09T12:58:04Z

task/generate-odcs-compose/0.1/generate-odcs-compose.yaml

This task is mirrored from another repo, so changes made directly will eventually be overwritten by changes made there.

Will ignore building manifest for this task then

yftacherzog · 2025-01-09T12:58:36Z

task/provision-env-with-ephemeral-namespace/0.1/provision-env-with-ephemeral-namespace.yaml

This task is mirrored from another repo, so changes made directly will eventually be overwritten by changes made there.

same for this task as well

yftacherzog · 2025-01-09T12:59:22Z

task/verify-signed-rpms/0.1/verify-signed-rpms.yaml

This task is mirrored from another repo, so changes made directly will eventually be overwritten by changes made there.

same for this task as well

tkdchen

Could you please describe the solution in the commit message?

hack/build-manifests.sh

tkdchen · 2025-01-13T04:30:05Z

hack/verify-manifests.sh

+            echo "Please run ./hack/build-manifests.sh and update your PR" >&2;
+            exit 1;
+        }
+    fi


Looks like ./hack/build-manifests.sh becomes a mandatory part of the development workflow when working with tasks and pipelines. Can you describe this? Probably inside the build-manifests.sh script itself at least.

Described it in the README change here. Do you think it is sufficient or need to be added to the build-manifests.sh script as well?

The content linked tells what need to do for task and pipeline authors, it is not a description to the whole solution. In my mind, the description could tell, for example, what the task and pipeline authors should do, what the maintainers should do, how the kustomized manifests are validated, whether this validation changes the build-definitions task and pipelines contribution workflow, even the build-and-push, etc.

Updated the README now. We don't need to change anything on the build-and-push script.

tisutisu · 2025-01-15T06:15:21Z

/retest

tisutisu · 2025-01-15T07:35:11Z

/retest

README.md

tkdchen · 2025-01-21T07:42:33Z

hack/build-manifests.sh

+# with the task name separated by a space, for example:
+# SKIP_TASKS="git-clone init"
+
+SKIP_TASKS="generate-odcs-compose provision-env-with-ephemeral-namespace verify-signed-rpms"


Why allowing to skip tasks and pipelines?

These tasks are mirrored from another repo, there is some automation which syncs them. If we change anything manually, that will be reverted. So skipped them.

hack/build-manifests.sh

tkdchen · 2025-01-21T08:05:51Z

hack/verify-manifests.sh

+            echo "Please run ./hack/build-manifests.sh and update your PR" >&2;
+            exit 1;
+        }
+    fi


The content linked tells what need to do for task and pipeline authors, it is not a description to the whole solution. In my mind, the description could tell, for example, what the task and pipeline authors should do, what the maintainers should do, how the kustomized manifests are validated, whether this validation changes the build-definitions task and pipelines contribution workflow, even the build-and-push, etc.

tisutisu force-pushed the kustomize-build branch from fdbe145 to 3033272 Compare January 7, 2025 08:08

tisutisu marked this pull request as ready for review January 7, 2025 08:11

tisutisu requested a review from a team as a code owner January 7, 2025 08:11

tisutisu force-pushed the kustomize-build branch 3 times, most recently from 29e718a to fe32af3 Compare January 7, 2025 12:45

chmeliik mentioned this pull request Jan 7, 2025

extend the sast-coverity-check CI task to support buildful scanning #1653

Open

chmeliik reviewed Jan 7, 2025

View reviewed changes

tisutisu force-pushed the kustomize-build branch from fe32af3 to c9884f3 Compare January 9, 2025 12:27

tisutisu requested review from ligangty, yma96, amisstea, avi-biton, gbenhaim, yftacherzog, Omeramsc and a team as code owners January 9, 2025 12:27

tisutisu marked this pull request as draft January 9, 2025 12:32

yftacherzog reviewed Jan 9, 2025

View reviewed changes

tisutisu force-pushed the kustomize-build branch 2 times, most recently from 55f6f38 to 04430c6 Compare January 13, 2025 03:51

tisutisu marked this pull request as ready for review January 13, 2025 03:53

tisutisu force-pushed the kustomize-build branch from 04430c6 to fb458d7 Compare January 13, 2025 04:06

tkdchen reviewed Jan 13, 2025

View reviewed changes

tisutisu force-pushed the kustomize-build branch from fb458d7 to 7aaed3c Compare January 15, 2025 05:28

tisutisu force-pushed the kustomize-build branch from 7aaed3c to f8972c2 Compare January 21, 2025 06:40

tkdchen reviewed Jan 21, 2025

View reviewed changes

tisutisu force-pushed the kustomize-build branch 2 times, most recently from d83973b to 7d22b8c Compare January 23, 2025 12:09

Validate kustomize build in CI

d3a6320

tisutisu force-pushed the kustomize-build branch from 7d22b8c to d3a6320 Compare January 24, 2025 08:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: validate kustomize build of tasks and pipelines #1792

feat: validate kustomize build of tasks and pipelines #1792

tisutisu commented Jan 7, 2025 •

edited

Loading

chmeliik Jan 7, 2025

tisutisu Jan 8, 2025

tisutisu Jan 10, 2025

yftacherzog Jan 9, 2025

tisutisu Jan 10, 2025

yftacherzog Jan 9, 2025

tisutisu Jan 10, 2025

yftacherzog Jan 9, 2025

tisutisu Jan 10, 2025

tkdchen left a comment

tkdchen Jan 13, 2025

tisutisu Jan 13, 2025

tkdchen Jan 21, 2025 •

edited

Loading

tisutisu Jan 22, 2025

tisutisu commented Jan 15, 2025

tisutisu commented Jan 15, 2025

tkdchen Jan 21, 2025

tisutisu Jan 22, 2025

tkdchen Jan 21, 2025 •

edited

Loading

feat: validate kustomize build of tasks and pipelines #1792

Are you sure you want to change the base?

feat: validate kustomize build of tasks and pipelines #1792

Conversation

tisutisu commented Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkdchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkdchen Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tisutisu commented Jan 15, 2025

tisutisu commented Jan 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkdchen Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

tisutisu commented Jan 7, 2025 •

edited

Loading

tkdchen Jan 21, 2025 •

edited

Loading

tkdchen Jan 21, 2025 •

edited

Loading