[request] service discovery

[sonicaghi]

feedback from a developer:

There is one really cool use-case for kong I see: providing service discovery for Mesos. Currently there are several solutions for this like mesos-dns, bamboo, haproxy script that comes with marathon or even my own marathoner. DNS-based solutions suffer from delayed updates of service map and haproxy-based solutions suffer from often reloads of haproxy. Old instances of haproxy stay alive for a while and eat up resources and pid table, spawning new instances requires creation of a new process. Deploying 100-instance app over a cluster of 120 machines really shows how inefficient things are.

Kong could be a really good solution for the problem, taking in account all the plugins, efficiency of nginx and lua scription and all other goodies. The only problem is that you cannot dynamically set upstreams, same problem with haproxy. When this is fixed, kong would shine for service discovery.

Take a look at zoidberg, combined with could it could be a good solution.

[sonicaghi]

Related to #157

[lucamaraschi]

👍

[snakelab]

👍

[maksimlikharev]

yes, ability to modify upstreams from plugin would be really beneficial, is it will allow discovery service integration

[Tieske]

I don't think I grasp the full request made here.

Currently Kong can already be used for service discovery by creating an upstream with several targets. Say you create an upstream (a Kong loadbalancer) by name my-service-v1. To this upstream you can add target entities; ip/port/weight combinations. So add for example 192.168.12.1:80 and 192.168.12.4:80.

If you then configure a route and a service, where the service has a hostname specified as my-service-v1, Kong will route the traffic to the 2 IP addresses configured.

The dynamic part is that new containers spinning up can be added to the my-service-v1 upstream by a call to the Kong management api. Either the container themselves do it, or the orchestration tool you use. Kong will start routing traffic there, without any reloads etc.

Does this cover this topic, or are there additional features regarding service discovery, that are missing?

[maksimlikharev]

@Tieske - in real cloud env. we usually have dedicated server registry/discovery service, like Eureka or Consul or Something. Kong upstreams/targets are not really a replacement, at least now, for the dedicated discovery service.

Upstreams/Hosts, I know it is based on nginx capability and I think it has only REST API, not sure if functional access is available.

What we're testing is following, global plugin scans "a registry" service, and updates/creates upstreams and hosts based on some logic and registry information, yes it does HTTP calls to http://localhost:.

First Question, is this approach sound to you?

Why I think this is important, it allows flexibility, support of the legacy infrastructure and allows to serve "dynamic" services that are produced in a cloud environment via Kong, including containers.

Also it allows automatic Red/Black and Canary deployment, you just specify some predefined tag in registry metadata and Kong plugin will provide correct weight for the host, allowing correct traffic.

Second Question, should be such methods, that are, manipulation of upstream/host to be part of PDK