Unable to Access Internet from Within Pods

[RhamaDaragaaz]

Environmental Info:
K3s Version:
k3s version v1.28.2+k3s1 (6330a5b)
go version go1.20.8

Node(s) CPU architecture, OS, and Version:
Linux utlxvti00390 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Currently just running one cluster and it's got a handful of pods running on it.

Describe the bug:
Pods seem to be incapable of connecting to the internet. When on the server I can access resources as it routes through a corporate proxy before reaching the intended destination. However, when I enter a pod and attempt to access external IP addresses, for example just www.google.com, it fails to load, timing out. (wget was using in this example).

Steps To Reproduce:
I installed K3s as an airgap install, most external resources are blocked if they don't get routed through our approved corporate resources in some fashion so manually downloading the image from an approved repository was what I did.
Everything seemed to be working fine, I just didn't have any pods running. So I downloaded some images via script (image repositories are all blocked, some magic shenanigans occur when using a moby script to download the images and build/load them for us). The images are loaded into docker and then exported using the save command to get a tar file that I can then load into K3s.
The images I'm running are a Selenium Hub and 5 replicas of a Selenium Node Chrome image. (I had to increase the default volume size for the K3s mount to do this).
I modified the selenium-hub-deployment deployment to be a LoadBalancer type instead of NodePort and exposed the ports needed to access the Selenium grid. Now able to access the grid via my external IP and the default Selenium port 4444, shows fine but no nodes are connected.
After some configuration of the nodes deployment, adding the http and https proxy, event bus host, subscribe and publish ports and such the nodes came online and connected to the grid, great.
Attempt to connect to the grid via my IDE and I can see that connection is successfully established to the hub and traffic is routed to the node, showing up on the grid as being utilized and I can connect via the camera option to see that a browser is successfully started on the node. However, the website I'm attempting to access fails to load.

This is where I've tried a lot of things based on research to resolve the problem:

• Connect to pod and wget www.google.com and it times out.
• Unable to nslookup, lacking tools and can't install it since it can't reach the internet, so download new image dnsutils and have problems with DNS giving bad gateways or can't connect to the API, bunch of errors. In the end I've updated the config and removed iptables (was using nftables) in favor of the default iptables included with K3s, and that seems to have stopped those errors. I can now resolve host names with nslookup in that pod both for my environment and I can even resolve www.google.com. However, still unable to connect to the internet.
• I've checked my k3s.service.env file, it has my http_proxy and https_proxy showing the correct proxy it should be using. Also made sure to add my pods CIDR (even though it's supposed to be done automatically?). Anyway it doesn't seem like the proxy settings getting propagated into the pod itself, as wget for www.google.com shows it tries to connect directly, whereas on my node it routes to the proxy first, then www.google.com and connects. Do I need to inject the proxy somehow? The workarounds I've seen seem to indicate it shouldn't be necessary or common to have to do this.
• I've looked into the resolv.conf, my node shows my company domain, two nameservers and a search with company namespaces. On my pod the resolv.conf just shows 10.43.0.10, which is the IP for my kube-dns/coredns. I'm not sure what these should be so maybe an avenue for further research?
• Checked out NetworkPolicy but from what I understand the egress should allow all traffic by default unless you want to configure it to deny all instead?
• firewalld is disabled, I don't have the nm-cloud thing either.
• Tried a tcpdump but it doesn't seem to capture my pod traffic, and can't run it on the pod due to being unable to install it, would need to find a docker image with it pre-installed or something to attempt it.

Really I'm just out of ideas of what to try next, I've exhausted most avenues of my research at this point and could really use some informed guidance.

Expected behavior:
I am able to access the internet from within my pods.

Actual behavior:
I'm unable to access the internet from within my pods.

Additional context / logs:
Just a really messed up corporate environment, many tools are inaccessible or cumbersome to obtain. On a VPN behind an external firewall and everything gets routed through a proxy as well somewhere in the mess.

[harsimranmaan]

You'll likely need to set HTTP_PROXY and HTTPS_PROXY vars in your Pod as environment variables.

[brandond]

In addition to configuring the http proxy environment variables in any pods that need to connect out to the Internet, you may also need to set up a resolv.conf for k3s that uses your corporate DNS servers as the resolvers, and configure k3s to use that via the --resolv-conf option. If k3s cannot find suitable upstream resolvers in your system configuration, it will default to using 8.8.8.8, which may be blocked in your environment.

[RhamaDaragaaz]

How is this done exactly? I've seen on other issues in my research that people are using a mutating webhook to inject these variables? Is this really necessary or is there a built in way to set the proxy values at the pod level?
Issues 194

Another issue mentions that it's not possible to set them at run time even though they're set in the k3s.service.env file, they don't get propagated to the pods. They say Kustomize or some other tool is required.
Issues 2983

I'll check on the resolv.conf file again and check out what my corporate DNS servers are if they're not already in there. Provided they're there on the node, how do I propagate those values to the pod as well as it's clearly utilizing a different resolv.conf file as the contents don't match?

[RhamaDaragaaz]

Coming back to this. I've tried setting environment variables in my deployment. This sets the HTTP_PROXY and HTTPS_PROXY variables within the pod (I can echo them) however they are not actually being used when I try to do something like wget. It tries to access the URL directly instead of routing to the proxy specified.

On the second response, my resolv.conf appears to be correct, but again the resolv.conf file within the pod points to just the coredns cluster IP and doesn't match anything outside in the cluster/node.

Is there more to just setting the environment variables in the deployment? Do you happen to have any examples of how this works I can reference by chance?

[RhamaDaragaaz]

For anyone coming back to this, the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables are apparently case sensitive. You need to specify them as http_proxy, https_proxy, and no_proxy for them to work.

[brandond]

There is no industry standard for the proxy env vars; which are supported and how they are used varies by language and application. A best practice would probably be to set both variants. curl's behavior is the de facto standard that most things emulate, see https://everything.curl.dev/usingcurl/proxies/env