Annotations For Ingress Service Discovery

[esatterwhite]

With larger systems moving towards de-centralized and eventually consistent configuration trying to manage a single static ingress resource can be rather difficult.

What would be a bit cleaner would be to annotate the Service resource to be added to a particular ingress rather than having to build an Ingress resource precompiled with all of the services. Similar to the way you might annotate services for Prometheus metrics scraping, I would be able to annotate a service to be picked up by a specific kong ingress deployment.

Instead of having to determine ingress rules upfront:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-kong
  rules:
  - host: api.kong.com
    http:
      paths:
      - path: /v2
        backend:
          serviceName: service-two
          servicePort: 80
      - path: /v1
        backend:
          serviceName: service-one
          servicePort: 80

  - host: debug.kong.com
    http:
      paths:
      - path: /v1
        backend:
          serviceName: ldaw
          servicePort: 80

We could keep the Service definition in the repository for the application that uses it allowing the ingress configuration to be de-coupled from the services. Additionally allowing the service configuration to be determined at run time makes automated deployments significantly easier and reduces the need for complex templating

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
  name: ingress-kong
spec:
  rules:
    discovery: kong

A QA team builds a service with a spec for debugging

# github.com/team-qa/http-debug-service
kind: Service
metadata:
  name: http-debug-service
  annotations:
    konghq.com/ingress.class: kong
    konghq.com/ingress.path: /
    konghq.com/ingress.name: ingress-kong
    konghq.com/ingress.host: debug.kong.com
    konghq.com/ingress.port: http
spec:
  ports:
  - port: 80
    name: http
    targetPort: 3000

Another team builds an application with a different deployment spec

# github.com/team-development-one/service-one
kind: Service
metadata:
  name: service-one
  annotations:
    konghq.com/ingress.class: kong
    konghq.com/ingress.path: /v1
    konghq.com/ingress.name: ingress-kong
    konghq.com/ingress.host: api.kong.com
    konghq.com/ingress.port: http
spec:
  ports:
  - port: 80
    name: http
    targetPort: 3000

A second development team builds an application which belongs to the same host name as service-one but on a different path

# github.com/team-development-two/service-two
kind: Service
metadata:
  name: service-two
  annotations:
    konghq.com/ingress.class: kong
    konghq.com/ingress.path: /v2
    konghq.com/ingress.name: ingress-kong
    konghq.com/ingress.host: api.kong.com
    konghq.com/ingress.port: http
spec:
  ports:
  - port: 80
    name: http
    targetPort: 3000

It additionally makes adding new services / removing old services from a production environment or changing their configuration much easier.

We can deploy a 1 kong ingress once, and never have to change, add or remove rules. The need for multiple ingress / kongingress resources is also greatly reduce. Teams can define their own services the way the want and add them selves to an ingress autonomously

[esatterwhite]

It would also alleviate problems such as those described in #1852

[mdeltito]

This would be a really useful "mode" of rule definition/discovery

The main thing that stands out as a challenge when moving this to runtime is dealing with host/port/path conflicts. How does that get surfaced, and what is the end result? This is maybe less straightforward on startup than it is at runtime when a conflicting service is discovered.

[esatterwhite]

I could see paths being a problem. But the port definitions would probably be picked up off of the service definition itself. I'm not sure that there could be a conflict there. Maybe I'm mis-understanding.

using the same host on multiple services would add them all to the same host rule of an ingress. I think the only potential conflict would be pathing at that point

[rainest]

This would probably be better-addressed upstream to Kubernetes SIG Network. We conform to the upstream specifications as best possible, and aren't directly responsible for their design (we do participate in SIG Network discussions, but the specifications are ultimately community consensus).

I'm not seeing exactly what use case this would solve, however--it seems like it's just moving the configuration of Ingress rules out of Ingress resources into Service resources (the annotations generally specify the same information you'd include in an Ingress rule, though it's unclear how they'd handle port differentiation). Since you still need to specify the same information you'd specify for an Ingress rule, it seems like the only net change you'd get is a de facto shift of RBAC permissions, where configuration currently gated by Ingress permissions (Ingress rules) is instead gated by Service permissions. That doesn't mesh well with the Kubernetes RBAC model of granting permissions based on API group, resource kind, and namespace, since Ingress permissions would effectively be split between the Ingress and Service kinds, which aren't in the same API group, whereas granting permissions to both Ingress and Service resources does not run afoul of the RBAC model.

I'd recommend just having users modify both Ingress and Service objects to build a complete chain of external HTTP routing configuration (Ingress) and internal cluster routing (Service), but if you did want to configure rules this way, I'd recommend writing a custom controller with permissions to read Services and edit Ingresses to add rules based on Service annotations--that'd allow you to configure rules in this fashion while still working cleanly with applications that are only aware of the standard specifications and their own vendor-specific annotations.

FWIW, KongIngress isn't intended as a replacement or alternative to Ingress (though the name rather unfortunately suggests it is): we created it initially to hold Kong configuration that didn't fit into any standard Ingress or Service fields (such as the Kong service connection timeout, which Kubernetes Services don't have any native field for). For the most part, we've replaced their functionality with Ingress and Service annotations, though they're still available for backwards compatibility and Kong configuration that doesn't have annotations yet (mostly upstream configuration).

[esatterwhite]

(the annotations generally specify the same information you'd include in an Ingress rule, though it's unclear how they'd handle port differentiation)

Thats already apart of the service resource spec which you'd likely have to introspect anyway - You could pick off relevant information from that as sane defaults.

It could just as well be another annotation that indicates the name of the service if that information isn't inherently available

  annotations:
    konghq.com/ingress.class: kong
    konghq.com/ingress.path: /v2
    konghq.com/ingress.name: ingress-kong
    konghq.com/ingress.host: api.kong.com
    konghq.com/ingress.port: http
    konghq.com/ingress.service: service-one

I'm not seeing exactly what use case this would solve

Service discovery + resolution with dynamic deployments / configuration. Not needing to re-deploy / apply ingress rules as the shape of the cluster evolves.

it seems like it's just moving the configuration of Ingress rules out of Ingress resources into Service resources

It is. Our configuration is de-centralized + eventually consistent. More importantly, it is resolved at run time on the target cluster. We run a set of controllers that uses templates + config maps as data sources. Its not Helm. Its not Kustomize. Every service deploys its own ingress rules (because it knows what they are) and we dynamically generate and we have ingress resource from all of that.

#repo one
kind: ConfigMap
metadata:
  name: service-one-ingress
data:
  ingress: |+
    {
      "api.test.com": [
        {"path": "/v1/one", "service": "service-one", "port": "80"}
      ]
    }

#repo two
kind: ConfigMap
metadata:
  name: service-two-ingress
data:
  ingress: |+
    {
      "api.test.com": [
        {"path": "/v1/two", "service": "service-two", "port": "80"}
      ]
    }

#repo three
kind: ConfigMap
metadata:
  name: service-two-ingress
data:
  ingress: |+
    {
      "api.test.com": [
        {"path": "/v1/three", "service": "service-three", "port": "80"}
      ]
    }

These config maps are also actually templated, They look like this

    - apiVersion: v1
      kind: ConfigMap
      metadata:
        name: ingress-rules-foo
        namespace: default
      data:
        ingress_kong_rules: |+
          {
            "{{{api-host-name}}}": [
              {"path": "/v1", "service": "foo", "port": 80}
            ]
          }

Where we don't even know the host name this service will be deployed under until it is deployed to a target environment. That is resolved from a different config map thats been deployed to the cluster.

If we disable the application in repo three The templates are also dropped from the cluster, The deployment is deleted, and the service is removed. We have quite a bit of control over that. The downside is we have to re-deploy Kong and its templates when new deployments / repositories are created because what defines the ingress resource and what defines the rules of the ingress resource are housed separately in micro services.

Doing this with multiple config maps which is very meta. It would be much less overhead and mental hoops if there were annotations to describe this information. Much like the Prometheus annotations. I don't update Prometheus every time I deploy a new applications with metrics, I just annotate the deployment / service.

It would fit much more nicely if there were a way to doing this with the ingress controller itself w/ a CRD that combined when the KongIngress + Ingress bits are trying to do separately

Annotation driven config is a fairly established pattern.

[esatterwhite]

If there is a way to dynamically add / remove ingress rules from Kong and I'm not seeing it, would love to hear better approaches / ideas

But config driven by annotations would fit nicely into the way many K8S native things work, and it would allow us to simplify quite a bit about how we have to deploy Kong.