feat: add chainsaw based e2e tests

[eddycharly]

What

This PR adds chainsaw based e2e tests (https://github.com/kyverno/chainsaw)

How

Added a couple of chainsaw tests and updated ci to run those tests.

Breaking Changes

NONE

[eddycharly]

Talked about that with someone at KubeCon today, let's discuss further tomorrow =)

[jonstacks]

@eddycharly, thank you for stopping by the booth. I still need to talk with the rest of our team, but I think we will end up taking this and gradually build up more chainsaw tests.

Just linking the product page for others on our team as I found it helpful: https://kyverno.github.io/chainsaw/latest/

[eddycharly]

Thanks @jonstacks take your time and i hope you will find chainsaw useful.
BTW the nix package is here.

[hjkatz]

Wow thank you @eddycharly for this addition! I'll be looking to try and use these tests today and see if we can get this merged.

e2e tests like these will really help us reach stability 😄

[hjkatz]

@eddycharly I would like to play around with this locally and submit some changes for nix. Though I'm not exactly sure how to collaborate on this branch... perhaps I create my own fork and submit PRs to your fork so we can update this PR for our main fork? 😂 seems like a lot but maybe that would work?

Thankfully it seems Github has thought of this: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests

Would you kindly enable this setting so we could collaborate?

[eddycharly]

Would you kindly enable this setting so we could collaborate?

@hjkatz i'll come to the booth with my laptop, will be easier ;)

[hjkatz]

I'm unfortunately not at kubecon - But I'm sure @jonstacks will love to help out.

FWIW I think you did have this setting enabled because I was able to push a commit to your fork's branch: https://github.com/ngrok/ngrok-operator/pull/506/commits (2nd commit at the bottom)

[Alice-Lilith]

Huh... yaml driven k8s tests. This is neat 👀

[eddycharly]

I think you did have this setting enabled because I was able to push a commit to your fork's branch

Ok, cool. I'll step by the booth anyway. Let me know if you have more questions.

[eddycharly]

Hi folks, i'm back at work. Let me know if you consider merging this PR and use chainsaw, i can help with changes if you want.
cc @jonstacks @hjkatz @Alice-Lilith

[hjkatz]

@eddycharly Sorry for the delay - We're are excited to merge this and I'll be focused on it today.

Q. Do you have any additions for the PR before we merge?

[eddycharly]

Do you have any additions for the PR before we merge?

Not really, next step would be to add more tests, maybe i will check the issues to see if they can be converted to a test easily.
In Kyverno user reporting issues usually contain manifests to reproduce the issue and it's often very easy to turn those into a test, maybe it's the same here 🤞

[eddycharly]

BTW we have a dedicated slack channel you can use if you have questions https://kubernetes.slack.com/archives/C067LUFL43U

[eddycharly]

@hjkatz either the pod doesn't start or keeps restarting because of invalid api-key/token 🤔
I can look at it later if it helps, you can also temporarily disable this test until you come up with a solution for the api-key/token.

[hjkatz]

hah, sorry for any emails - We're iterating on this right now trying to figure out exactly how we want these to run.

[hjkatz]

@eddycharly Sorry for all of the crazy noise! This is finally ready to merge. There were some adjustments @jonstacks and I decided to make so the e2e tests will work for both ngrok-op/main and forks. tl; dr:

• Separate out changes and build-and-test into composite github actions
• Create 2 workflows: ci.yaml (pull_request) and test.yaml (pull_request_target)
• Ensure that each workflow has access to secrets (or not) as needed

[eddycharly]

@hjkatz @jonstacks that's awesome !

I hope you like chainsaw, don't hesitate to ask questions on our slack channel if needed or file issues or contribute to the project if you want.

Additionally you can add yourself in the ADOPTERS file if you want.

I didn't have much time to help but i will try to search the issues and see what tests it makes sense to add when i have more time.