Consider dropping Traefik as a bundled component

[ChristianCiach]

Well, this will probably be a controversial topic, but I've given this a lot of thought and I think it is at least worth talking about.

I know that the deployment of the bundle Traefik can be disabled via an installation option. Please don't disregard the following just because of this fact.

Also, I know that my comments sometimes come across as aggressive. This is not my intention at all. Please understand that this discussion comes from a place of love.

After heavy usage of K3s for approximately two years to deploy a ton of clusters, I think Traefik is not a very good fit as a bundled component. These are my thoughts in no specific order:

• While the other bundled components are either de-facto standard components for Kubernetes with (if at all) up to two options (containerd, CoreDNS) or are developed or at least maintained by Rancher/Suse ifself (Local-path-provisioner, Flannel), this cannot be said about Traefik. Traefik is one of many, many ingress controllers.
• Adding to the last point, there are probably many other more open, community-driven ingress controllers that are more lightweight (without adding a lot of CRDs) and less driven by commercial interests.
• Traefik phones home by default (if not disabled via --global.checknewversion). The update-check doesn't even make sense when using it as a bundled k3s component.
• Traefik continues to advertise their proprietary services in the open source distribution, without providing an option to turn these links off. They aggressively defend this against the explicit wishes of many users:
  • Discard Traefik Pilot notifications in the UI traefik/traefik#7360
  • Config option to remove Hub button from dashboard traefik/traefik#9108
• K3s has a bad track record in keeping up with the current Traefik versions. The current bundled version is v2.6.2, which isn't even supported in any way: According to https://doc.traefik.io/traefik/deprecation/releases/ , Traefik 2.6 reached EOL on May 24, 2022. While K3s also sometimes lags behind a bit when it comes to some other bundled components, these are at least always reasonably up to date.
• Traefik is the only bundled component that is deployed by the bundled Helm Controller and thus is the only component customizable via HelmChartConfig. While I would be happy to be able to customize the other components this way, this is still an outliner. A point could be made that the bundled Helm controller would become obsolete without the bundled Traefik.
• Traefik feels like the least essential of the bundled components. It is very rare to see clusters without a DNS server or a CNI plugin, but many clusters don't need an ingress controller. I am sure many people deploy K3s with its default settings and don't even notice the bundled Traefik. Worse: If they eventually decide to install another ingress controller, they first need to re-install K3s to disable the deployment of the bundled Traefik. (I don't even know if setting --disable traefik on a running cluster removes an already installed Traefik).

I still think bundling an ingress controller could be worthwhile, but this inevitable forces you to "play favorites" - way more so than with any of the other bundled components. Even when bundling an ingress controller, I still think there are better, less tainted options (maybe something from under the CNCF umbrella).

[ChristianCiach]

I am not sure if this is a proper fit for a "discussion". I can convert it to an issue if you want. Truth be told: I kinda wanted this discussion to be referenced in traefik/traefik#9108 , but this only seems to work with issues, not discussions.

[brandond]

Just to respond to these comments in order:

While the other bundled components are either de-facto standard components for Kubernetes with (if at all) up to two options (containerd, CoreDNS) or are developed or at least maintained by Rancher/Suse ifself (Local-path-provisioner, Flannel), this cannot be said about Traefik. Traefik is one of many, many ingress controllers.

Exclusively or preferentially using projects maintained by SUSE is not one of our goals. K3s is a CNCF project, and we aim to keep it as open and independent as possible.

Adding to the last point, there are probably many other more open, community-driven ingress controllers that are more lightweight (without adding a lot of CRDs) and less driven by commercial interests.

What would you suggest? Nginx can hardly be held up as example of an open-source project that's "less driven by commercial interests". While the ingress controller is a community project, nginx itself has a closed development process.

Traefik phones home by default (if not disabled via --global.checknewversion). The update-check doesn't even make sense when using it as a bundled k3s component.

This is a valid concern, but not one I see expressed very often.

Traefik continues to advertise their proprietary services in the open source distribution, without providing an option to turn these links off. They aggressively defend this against the explicit wishes of many users:
traefik/traefik#7360
traefik/traefik#9108

This sounds like something you have a personal philosophical objection to, but I don't see that it affects the core functionality of Traefik as an ingress controller.

K3s has a bad track record in keeping up with the current Traefik versions. The current bundled version is v2.6.2, which isn't even supported in any way: According to https://doc.traefik.io/traefik/deprecation/releases/ , Traefik 2.6 reached EOL on May 24, 2022. While K3s also sometimes lags behind a bit when it comes to some other bundled components, these are at least always reasonably up to date.

This is valid. We do a better job of keeping up with ingress-nginx on the RKE/RKE2 side because it's been affected by so many critical CVEs. Can't say the same for traefik...

Traefik is the only bundled component that is deployed by the bundled Helm Controller and thus is the only component customizable via HelmChartConfig. While I would be happy to be able to customize the other components this way, this is still an outliner. A point could be made that the bundled Helm controller would become obsolete without the bundled Traefik.

We would like to increase, not decrease, the number of things that can be customized via HelmChartConfig. For example, it is very difficult to customize the coredns configuration at the moment; bundling it as a helm chart would make the user experience much better in that regard.

Traefik feels like the least essential of the bundled components. It is very rare to see clusters without a DNS server or a CNI plugin, but many clusters don't need an ingress controller. I am sure many people deploy K3s with its default settings and don't even notice the bundled Traefik. Worse: If they eventually decide to install another ingress controller, they first need to re-install K3s to disable the deployment of the bundled Traefik.

We are intentionally opinionated about what we include. To paraphrase from our README, K3s "packages additional components and services necessary for a fully functional cluster that go beyond vanilla Kubernetes." Leaving out an ingress controller would not accomplish this goal.

(I don't even know if setting --disable traefik on a running cluster removes an already installed Traefik).

It does.

[ChristianCiach]

Thank you for engaging this topic! I am actually very satisfied with your answers, because I didn't actually expect K3s to drop Traefik. After all, "considering it" was everything that I asked for :)

To address some of your answers:

Exclusively or preferentially using projects maintained by SUSE is not one of our goals. K3s is a CNCF project, and we aim to keep it as open and independent as possible.

I applaud that! As long as the chosen components are a good "fit" for K3s (however one may define that) and they are actively maintained, I am fine either way.

What would you suggest? Nginx can hardly be held up as example of an open-source project that's "less driven by commercial interests". While the ingress controller is a community project, nginx itself has a closed development process.

Yes, that is true. I don't really have a better alternative for you. Traefik and Nginx-Ingress are actually the only ingress controllers I have any experience with. Googling "cncf ingress controller" brings up Contour, but I've never used that.

This sounds like something you have a personal philosophical objection to, but I don't see that it affects the core functionality of Traefik as an ingress controller.

Yes, that is a purely personal philosophical objection, for sure! I won't deny that :) But let be elaborate about that: I have no issue with paying for someware (I do that all the time) and I have no problem when software development is driven by commercial interests. The last part is true for pretty much all products in the Kubernetes ecosystem. It would be hypocritical for me to claim otherwise, since I've been a software developer (mostly Java) for well over 10 years before having switched to DevOps about three years ago.

What I react negatively to is when the open source editions of products (sometimes inappropriately called "community edition") are used to show "advertisements" (in the loosest sense of the word) to the users to push the proprietary or commercial products of the company. Take Nginx as an example: Yes, there is "Nginx Plus" and this is fine, but I've never seen any message in any management UI of the open source Nginx distribution that pushes me to use Nginx Plus. Going back to Contour, which seems to be made by VMware: As I said, I don't know anything about Contour, but we are heavily using other products made by VMware, like kapp, kbld, imgpkg, Harbor registry... VMware seems to be very good at keeping their open source and commercial offerings separate (as is Suse :) ). Not once has kapp asked me to switch my cluster to VMware Tanzu.

Traefik on the other hand seems to be eager to use their open source edition to promote their proprietary services. I would maybe be able to accept that. But when they tried to push Traefik Pilot, the dashboard of Traefik was using an IFrame to load stuff from their servers, and these requests included possibly identifying information about the Traefik instance. This is were I cross the line.

This is valid. We do a better job of keeping up with ingress-nginx on the RKE/RKE2 side because it's been affected by so many critical CVEs. Can't say the same for traefik...

It is true that Traefik itself rarely shows up in CVE scans, but the same is not true for the included Go modules. For example, when scanning Traefik 2.6.2 using Trivy, it reports this:

traefik:v2.6.2 (alpine 3.14.5)

Total: 5 (UNKNOWN: 0, LOW: 0, MEDIUM: 2, HIGH: 2, CRITICAL: 1)

usr/local/bin/traefik (gobinary)

Total: 19 (UNKNOWN: 7, LOW: 1, MEDIUM: 4, HIGH: 7, CRITICAL: 0)

But even if this wasn't the case: Some of our customers demand SBOM exports of all software components used in the cluster, so they will ask us about the outdated versions. To avoid that, we currently disable the bundled Traefik of K3s and just deploy it ourselves, but using newer versions :)

We would like to increase, not decrease, the number of things that can be customized via HelmChartConfig. For example, it is very difficult to customize the coredns configuration at the moment; bundling it as a helm chart would make the user experience much better in that regard.

Agreed! I was actually a bit sad when I saw the latest changes to be able to add more customization to the CoreDNS configuration didn't use the HelmChartConfig mechanism.

---

Let me take this opportunity to say THANK YOU for all your efforts engaging with the community! I have been lurking this Github project (commits, releases, issues, discussions) for about two years now and your way to address all of the issues and discussions deserves a lot of respect! You always seem to be eager to help, even when the issues or questions are clearly by people new to K3s, or when the questions are actually about Kubernetes and not specific to K3s, or even when the issues can be tracked down to unsupported custom builds of K3s. I applaud your patience and your readiness to invest huge amounts of your time helping all of us out!