🌱 Configure ENVTEST Binaries for IDE Debugging

[camilamacedo86]

This change introduces downloading the specific binaries required to run ENVTEST-based tests into the project/bin directory. This setup makes configuring tests for debugging directly in an IDE easier.

Motivation

1. Ease of IDE Debugging:
   • By setting the BinaryAssetsDirectory dynamically, contributors can debug tests in their preferred IDEs (e.g., GoLand, VSCode) without requiring additional environment configuration such as setting KUBEBUILDER_ASSETS.

2. Project-Specific Binaries:

Installing binaries inside the project avoids conflicts with other projects or system-wide installations, particularly useful for contributors with restricted environments (e.g., corporate laptops).

The binaries installed by envtest-setup are no longer placed in the HOME/global directory. See the default behavior here:
https://github.com/kubernetes-sigs/controller-runtime/blob/e3347b5405cdd0da5bff527af3d406117938ba6b/tools/setup-envtest/README.md?plain=1#L57-L70

This change ensures binaries are placed locally, avoiding the conflicts and clutter caused by global installations.

I’m referring to the binaries required for ENVTEST (not those managed by BINGO).

Before this PR those are placed for example in a Mac OS at: /Users/camilam/Library/Application Support/io.kubebuilder.envtest/k8s/1.31.0-darwin-arm64

$ pwd
/Users/camilam/Library/Application Support/io.kubebuilder.envtest/k8s/1.31.0-darwin-arm64

ls -la
total 337856
dr-xr-xr-x  5 camilam  staff       160 19 Nov 09:20 .
drwxr-xr-x  3 camilam  staff        96 19 Nov 09:20 ..
-r-xr-xr-x  1 camilam  staff  26058562 19 Nov 09:20 etcd
-r-xr-xr-x  1 camilam  staff  90359954 19 Nov 09:20 kube-apiserver
-r-xr-xr-x  1 camilam  staff  56560754 19 Nov 09:20 kubectl

3. Simplified Contributor Onboarding:
   • New contributors often face a learning curve to:
     • Understand ENVTEST and its dependency on binaries.
     • Configure KUBEBUILDER_ASSETS manually.
     • Set up their IDE for debugging tests.
   • This setup minimizes friction and ensures tests can be run consistently across different environments.

While setting KUBEBUILDER_ASSETS is sufficient for experienced contributors, this change makes it easier for newcomers and supports debugging directly in IDEs without additional configuration.

[netlify]

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	7f4439e
🔍 Latest deploy log	https://app.netlify.com/sites/olmv1/deploys/673cb3a338aeb50008528d71
😎 Deploy Preview	https://deploy-preview-1454--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[camilamacedo86]

We are updating the setup to use the project's bin directory (used already for the other staff), ensuring we don’t interfere with contributors' local development environments."

[joelanford]

iirc, the reason this is here is because Red Hat reported issues when they ran CI because of quirks of how their CI system sets up the environment. We would be good stewards if we validated changes in this area in RH's CI system before we make them here.

[joelanford]

Note the conditional: in order for this override to be used we need both:

• $HOME=="" or $HOME=="/"
• XDG_DATA_HOME==""

That will not happen in a developer's local environment.

[camilamacedo86]

Thanks for share why it is there

My main idea for local development is to keep the binaries within the project. This way, I can configure ENVTEST properly and debug the tests using the IDE.

I'll take a look into this.

I also don’t want the targets to install anything in my Go workspace or anywhere else outside of the project.

[camilamacedo86]

@joelanford Is not only used because the default path will be /usr/local/kubebuilder/, which does not work in the GitHub/ci. If we move to the project's bin dir, would the issues be reported?

[m1kola]

Unresolving because I have a related question.

WIth --bin-dir set to a local directory - do we need a conditional for /tmp/envtest-binaries?

[camilamacedo86]

In my opinion, we don't need /tmp/envtest-binaries anymore. However, Joe raised concerns about this change, and I agree it's not worth risking breaking downstream CI for now.

Why POV it's no longer needed:

Previously, envtest-setup installed binaries in $HOME, which caused issues with GitHub Actions and other CI systems. With the current changes, binaries are now placed in project/bin, resolving those issues. This makes the temporary /tmp/envtest-binaries logic unnecessary.

However, that is not the goal of the PR, we can address it in a follow up

That said, I don't want to hold up this PR based on Joe's concerns. It's not the focus of this PR.
We can address this in a follow-up PR to simplify CI by removing the now-unnecessary logic for placing envtest binaries in /tmp/.

[camilamacedo86]

@m1kola

Is that make sense ?
Can we close this one?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.69%. Comparing base (e5820ae) to head (7f4439e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1454      +/-   ##
==========================================
- Coverage   74.73%   74.69%   -0.04%     
==========================================
  Files          42       42              
  Lines        3241     3241              
==========================================
- Hits         2422     2421       -1     
- Misses        646      647       +1     
  Partials      173      173              

Flag	Coverage Δ
e2e	52.32% <ø> (-0.10%)	⬇️
unit	57.17% <ø> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[camilamacedo86]

Hi @m1kola,

Thank you for catching that—really good observation! 🙌

Your comment highlights an additional motivation for this change: addressing scenarios like those outlined in kubernetes-sigs/controller-runtime#2744 and kubernetes-sigs/controller-runtime#2646.

The required changes have been addressed, and we no longer use bingo here as intended. Please feel free to review this one.

[joelanford]

I like the integration in TestMain to make envtest work for IDE-based test execution. Could we put that piece in its own PR?

[camilamacedo86]

Hi @joelanford, @m1kola, @perdasilva, @bentito,

Apologies for the confusion earlier!

This PR is NOT about Bingo. The main goal is to make debugging ENVTEST-based tests in the IDE simpler and more straightforward.

I suspect that some team members might think it’s not possible to debug with Ginkgo because the tests in question are actually ENVTEST-based. This change addresses that scenario and clarifies the process.

I’ve addressed all comments, closed all as done or outdated to avoid confusion, reverted the BINGO changes, and kept the suggestions.

@m1kola, could you please revert your block?

Thanks in advance! 😊

[camilamacedo86]

Hi @perdasilva,

You added this comment, and I deviated—sorry about that!

Would it be cleaner if we just used an environment variable like ENV_TEST_BINARY_DIR or something like that?
Then, it doesn't really matter where the binaries are placed.

We already have an environment variable (KUBEBUILDER_ASSETS) that can be used, but it’s not very convenient and involves a steep learning curve:

• Know that ENVTEST requires binaries.
• Then, by default, ENVTEST is expected to find the bins in local/....
• We can override this using the KUBEBUILDER_ASSETS environment variable.
• Note that those bins never was in the $GOPATH/bin because they are placed via the setup-invest in the global place (see here) which indeed cause issues for those which have restrictive envs.
• Then, there’s the added challenge of knowing how to configure the tests properly in an IDE.

This PR simplifies the process so everything “just works” without extra setup or deep knowledge.
Making tests and debugging smoother and hassle-free.

I’ve also updated the code comments to clarify things further.
Hope this clears everything up! 😊

[camilamacedo86]

Hi @m1kola,

Just to clarify, the binaries installed by envtest-setup are no longer placed in the HOME/global directory. See the default behavior here:
https://github.com/kubernetes-sigs/controller-runtime/blob/e3347b5405cdd0da5bff527af3d406117938ba6b/tools/setup-envtest/README.md?plain=1#L57-L70

This change ensures binaries are placed locally, avoiding the conflicts and clutter caused by global installations.

To clarify, I’m referring to the binaries required for ENVTEST (not those managed by BINGO).

Hope this clears things up! 😊

[camilamacedo86]

@perdasilva ^ That might clarify better.

[m1kola]

I don't think this needs to change: to put binaries produced by setup-env into a separate directory - updating SETUP_ENVTEST_BIN_DIR_OVERRIDE above is enough.

[camilamacedo86]

The goal of the change is to simplify.
Make it easier to figure out that the tests use KUBEBUILDER_ASSETS to set the path for the bins.

[m1kola]

I'm on the fence here because of repetition: we would have to now ensure that we pass the same args to both $(SETUP_ENVTEST) calls. But explicit is better than implicit I guess.

So I'll leave it up to you.

[m1kola]

Unresolving because I have a related question.

WIth --bin-dir set to a local directory - do we need a conditional for /tmp/envtest-binaries?

[camilamacedo86]

Hi @m1kola

Thank you. Regards #1454 (comment) see : #1454 (comment)

That is not the goal of the PR, and this point can be addressed in a follow-up.

[m1kola]

Looks good to me overall.

Just one question about $(SETUP_ENVTEST) and setup-envtest targets.

[m1kola]

I'm on the fence here because of repetition: we would have to now ensure that we pass the same args to both $(SETUP_ENVTEST) calls. But explicit is better than implicit I guess.

So I'll leave it up to you.

[m1kola]

A bit strange to have $(SETUP_ENVTEST) and setup-envtest targets.

Is it beneficial to have a separate setup-envtest target here?

Maybe we can just do this?

Suggested change

	test-unit: $(SETUP_ENVTEST) setup-envtest #HELP Run the unit tests
	test-unit: $(SETUP_ENVTEST) #HELP Run the unit tests
	mkdir -p $(ROOT_DIR)/bin
	$(SETUP_ENVTEST) use -p env $(ENVTEST_VERSION) $(SETUP_ENVTEST_BIN_DIR_OVERRIDE)