Replace redis with consul

[jbardin]

Use Consul for configuration and service registration.

• Unify runtime configuration and app configuration into a single file. Each config will have everything needed to run a single service, including docker configuration and app environment variables. Store this as a single document in Consul, and monitor for changes.
• Use Consul's service registration for... service registration. The first step will be to use the API to register services with TTL checks, which mimics our current implementation. We could then later choose maintain registration via other health checks if needed.
• Services will still use shuttle to connect to other services, and forgo discovery on their own. The new config will allow /etc/hosts entries to be defined, so that *.galaxy hosts can still be used.
• Each service host will be running a consul agent, and all consul operations will happen against 127.0.0.1.
• consul servers will be bootstrapped onto 3 static hosts, and the addresses will be provided in the initial configuration for all service hosts.
• Port mappings will allow service to define more than one port to be handled through shuttle

[joeshaw]

I'm not wild about the /etc/hosts thing, because it eliminates our ability to do things dynamically and introduces cross-service ordering concerns. I'd rather we just punt on the .galaxy DNS thing for now (since everything routes through shuttle currently) and only set up an /etc/hosts entry for the currently running service (for VIRTUAL_HOST_502 (etc) values, which do require a non-load-balanced endpoint).

Everything else 👍 sounds good though.

[jbardin]

The proposed use of /etc/hosts here is specifically just to route to shuttle, nothing more. It could allow containers to use *.galaxy to connect to the local shuttle instance using a hostname, working exactly as he have it now.

We can certainly choose to only use it for VIRTUAL_HOST_502 et al, and keep dnsmasq. I'd just like it in the config only because it could still prove useful at some point, and it's easy to implement since it's a feature directly supported by docker.

[joeshaw]

I think /etc/hosts works great for VIRTUAL_HOST_502, and that's currently our only use of the .galaxy domain. So I don't think that dnsmasq is needed to be kept around for that. Although isn't there a chicken-and-egg problem with routing for shuttle for that? shuttle itself is what uses VIRTUAL_HOST_502 to connect to the real service instance to get its contents, right?

[jbardin]

Now that I've got a better handle on how things work, it looks like galaxy features can map to consul like so

• App Configuration -> Key/Value storage: All parameters for an app can be kept in a single blob in KV storage. That key then can be "watched" for changes
• Service Registration -> Service Catalog: The service api is more limited than what we use for service registration, but some of what we have may not be needed, and other features may bet better moved to another aspect based on sessions.
• Notifications -> Events: Consul has an events api which can be used to trigger deploys, restarts, etc.
• Limits -> Sessions: sessions can be used to create locks/semaphores that can be used to limit number of restarting nodes.
• Runtime Info -> Sessions: Sessions may also be used to set a TLL for stored values, so that transient runtime information can be stored and allowed to timeout without the need for cleanup.