Both Endpoints and EndpointSlices not working with Service abstraction for external IPs

[PMacho]

Environmental Info:

K3s Version:
k3s version v1.28.7+k3s1 (051b14b)
go version go1.21.7

Node(s) CPU architecture, OS, and Version:

Linux kubernetes-test-server 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 server, 2 agents, Traefik disabled, using Nginx instead, almost clean install

Describe the bug:

As recommend in https://kubernetes.io/docs/concepts/services-networking/service/ I did try to abstract access to an externally hosted mariaDB. However, both the recommend approach and the legacy approach with Endpoints did not work as expected. There is a slight difference however, in the EndpointSlice case the service description showed Endpoints: <none> while in the endpoints case it was empty, i.e. Endpoints: .

Steps To Reproduce:

It is a clean K3s with --disable traefik, nginx ingress as well as cert-manager installed via helm, plus GitLab Agent for Kubernetes also using Helm.

The Service:

apiVersion: v1
kind: Service
metadata:
  name: mariadb
spec:
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306

I've tried both, EndpointSlice:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: mariadb-1
  labels:
    kubernetes.io/service-name: mariadb
addressType: IPv4
ports:
  - name: mysql
    port: 3306
    protocol: TCP
endpoints:
  - addresses:
    - 10.0.0.5

, and Endpoints:

apiVersion: v1
kind: Endpoints
metadata:
  name: mariadb
subsets:
  - addresses:
      - ip: 10.0.0.5
    ports:
      - name: mysql
        port: 3306

Both were successfully deployed in the same namespace as the service and linked to the service.

case EndpointSlices

kubectl get endpointslice -l kubernetes.io/service-name=mariadb

NAME        ADDRESSTYPE   PORTS   ENDPOINTS   AGE
mariadb-1   IPv4          3306    10.0.0.5    3m29s

kubectl describe service mariadb

Name:              mariadb
Namespace:         keldysh-projects-workshop
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: service
                   meta.helm.sh/release-namespace: keldysh-projects-workshop
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.126.23
IPs:               10.43.126.23
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

case Endpoints

kubectl describe endpoints mariadb:

Name:         mariadb
Namespace:    keldysh-projects-workshop
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: service
              meta.helm.sh/release-namespace: keldysh-projects-workshop
Subsets:
  Addresses:          10.0.0.5
  NotReadyAddresses:  <none>
  Ports:
    Name   Port  Protocol
    ----   ----  --------
    mysql  3306  TCP

Events:  <none>

kubectl describe service mariadb

Name:              mariadb
Namespace:         keldysh-projects-workshop
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: service
                   meta.helm.sh/release-namespace: keldysh-projects-workshop
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.126.23
IPs:               10.43.126.23
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         
Session Affinity:  None
Events:            <none>

Expected behavior:

The external database should be reachable through the service abstraction.

Actual behavior:

My applications just only reach the empty service.

Additional context / logs:

DNS is working in principle:

kubectl exec -i -t dnsutils -- nslookup mariadb

Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   mariadb.keldysh-projects-workshop.svc.cluster.local
Address: 10.43.19.67

[brandond]

My applications just only reach the empty service.

Have you read: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

You can create what are termed headless Services, by explicitly specifying "None" for the cluster IP address (.spec.clusterIP).
You can use a headless Service to interface with other service discovery mechanisms, without being tied to Kubernetes' implementation.

It looks like you're trying to use a headless service, but appear to have missed the bit where you set spec.clusterIP: None.

[PMacho]

I did read the section about headless services. However, it explicitly says:

Sometimes you don't need load-balancing and a single Service IP. In this case, you can create what are termed headless Services, ...

I cannot say, I do not need load-balancing. Of course, in case of a single MariaDB instance that might apply but in general I certainly want load-balancing. Therefore, I did not want to use a headless service.
What I am trying to use is a "Service without selectors":

https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors

... also described for example in

https://www.googblogs.com/kubernetes-best-practices-mapping-external-services/
https://theithollow.com/2019/02/04/kubernetes-endpoints/

The official kubuernetes documentation on "Service without selectors" says:

Services most commonly abstract access to Kubernetes Pods thanks to the selector, but when used with a corresponding set of EndpointSlices objects and without a selector, the Service can abstract other kinds of backends, including ones that run outside the cluster.

For example:

• You want to have an external database cluster in production, but in your test environment you use your own databases.

This is what I want to achieve.

None of above sources, including the official documentation, say that this only works with headless services. The example in the official documentation is:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

and

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: my-service-1 # by convention, use the name of the Service
                     # as a prefix for the name of the EndpointSlice
  labels:
    # You should set the "kubernetes.io/service-name" label.
    # Set its value to match the name of the Service
    kubernetes.io/service-name: my-service
addressType: IPv4
ports:
  - name: '' # empty because port 9376 is not assigned as a well-known
             # port (by IANA)
    appProtocol: http
    protocol: TCP
    port: 9376
endpoints:
  - addresses:
      - "10.4.5.6"
  - addresses:
      - "10.1.2.3"

For the record, I just tried defining the mariadb service being headless with the EndpointSlices appraoch. The routing indeed works. The description of the service however is still not aware of what it actually does, i.e. "Endpoints: ":

kubectl describe service mariadb

Name:              mariadb
Namespace:         keldysh-projects-workshop
Labels:            app.kubernetes.io/managed-by=Helm
Annotations:       meta.helm.sh/release-name: service
                   meta.helm.sh/release-namespace: keldysh-projects-workshop
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  RequireDualStack
IP Families:       IPv4,IPv6
IP:                None
IPs:               None
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

[brandond]

My applications just only reach the empty service.
DNS is working in principle:

kubectl exec -i -t dnsutils -- nslookup mariadb

Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   mariadb.keldysh-projects-workshop.svc.cluster.local
Address: 10.43.19.67

If the DNS lookups return the correct address, and the endpoints return the correct address, what exactly is your application looking at to find the service? This sounds like perhaps a problem with your application? Regardless this doesn't sound like an issue with K3s.

[PMacho]

Well, saying DNS is working in principle, I mean, it does resolve something, i.e. the Service's ClusterIP. This is an address but not the correct one. According to the documentation (https://kubernetes.io/docs/concepts/services-networking/service/#service-no-selector-access) above lookup should rather return 10.0.0.5, i.e. the IP defined in the EndpointSlices.

I submitted this as a bug in K3s since the example is almost a copy-past of the official kubernetes documentation but does not behave accordingly.

[brandond]

We don't modify any of the core Kubernetes behavior. If the upstream documentation examples don't do what you're expecting, you might consider opening an issue with the upstream project.

I will say that it seems to work for me:

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: default
spec:
  clusterIP: None
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      name: http
    - protocol: TCP
      port: 443
      targetPort: 443
      name: https
---
apiVersion: v1
kind: Endpoints
metadata:
  name: httpbin
  namespace: default
subsets:
  - addresses:
      - ip: 44.217.139.104
    ports:
      - name: http
        port: 80
      - name: https
        port: 443
---
apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3
    command:
      - sleep
      - "infinity"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

brandond@dev01:~$ kubectl exec -i -t dnsutils -- dig httpbin.default.svc.cluster.local.

; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> httpbin.default.svc.cluster.local.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57732
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;httpbin.default.svc.cluster.local. IN	A

;; ANSWER SECTION:
httpbin.default.svc.cluster.local. 5 IN	A	44.217.139.104

;; Query time: 0 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Thu Mar 14 18:06:01 UTC 2024
;; MSG SIZE  rcvd: 111

I will note that the IP you got back from your nslookup test isn't the same as the clusterIP for the service. Was that a typo, or do you have some other wildcard record in one of your search domains that is masking the lookup when you do not fully qualify it?

[PMacho]

As noted above, using a headless service, i.e setting clusterIP: None, did work for me too. But this was not the goal as it also removes kubernetes capabilities of load-balancing.
I will consider opening an issue in the upstream project. Thank you for your time.

[brandond]

It is my understanding that load balancing for ClusterIP services only works if the service is backed by pods within the cluster. I don't believe it is expected that kube-proxy will load-balance to things outside the cluster. These are good questions to ask of the core Kubernetes maintainers though, if there is any ambiguity.

[PMacho]

In my understanding, if this is not intended the kubernetes documentation is quite misleading here. Additionally, in the given case it even explicitly lists abstraction of external databases as a common use case for Services without Selectors.

FYI, I just created a bug report in the official repo: kubernetes/kubernetes#123954

[PMacho]

In case anyone else also encounters the same issue: The reason it did not work where the port names, i.e. the Service and the EndpointSlices ports' names need to match in order for kube-proxy to pick them up correctly. The kubernetes docs are being updated accordingly, reference: kubernetes/website#45579 .

[brandond]

Glad you were able to figure it out, and make some improvements to the docs. I definitely haven't made much use of headless services or manually-maintained endpoint lists; the state of the docs suggests that not many other folks do either.