Talaria rehasher is dropping all the WS connections when a new talaria instance is added in consul.

[Sachin4403]

Hi Team,

I am using Kratos to create the WS connection with a talaria instance called talaria0. In the end, talaria0 will have 3000 WS connections. now i am adding 3 new talaria instances in consul. ideally, rehasher should rebalance it like 750 WS at each talaria but whenever we are receiving an event from the consul about new talaria instances it is dropping all of the connections.

We are using consul 31.1 version with server enabled https://github.com/hashicorp/consul-helm/blob/6a1a2d3cd5d69b3a3f8109dce3d4663089559d0b/values.yaml#L301 and UI enabled https://github.com/hashicorp/consul-helm/blob/6a1a2d3cd5d69b3a3f8109dce3d4663089559d0b/values.yaml#L1022

Talaria version v0.5.11
Talaria Configuration

fqdn: talaria
env: dev
scheme: http
primary:
  address: :6200
health:
  address: :6201
pprof:
  address: :6202
control:
  address: :6203
metric:
  address: :6204
  metricsOptions:
    namespace: "xmidt"
    subsystem: "talaria"
log:
  file: "stdout"
  level: "debug"
  json: true
deviceAccessCheck:
  type: "monitor"
  sep: ","
  checks:
    -
      name: "Devices with trust level > 999"
      deviceCredentialPath: "trust"
      inputValue: 999
      op: gt
device:
  manager:
    upgrader:
      handshakeTimeout: "1s"
    maxDevices: 100000
    deviceMessageQueueSize: 100
    pingPeriod: "45s"
    idlePeriod: "135s"
    requestTimeout: "127s"
  outbound:
    method: "POST"
    eventEndpoints:
      default: http://xmidt-caduceus:6000/api/v3/notify
    requestTimeout: "125s"
    defaultScheme: "http"
    allowedSchemes:
      - "http"
      - "https"
    outboundQueueSize: 1000
    workerPoolSize: 100
    transport:
      maxIdleConns: 0
      maxIdleConnsPerHost: 100
      idleConnTimeout: "120s"
    clientTimeout: "160s"
    authKey: *********
inbound:
  authKey: *********
jwtValidator:
  keys:
    factory:
      uri: "http://xmidt-themis:6500/keys/dev"
    purpose: 0
    updateInterval: 604800000000000
service:
  defaultScheme: http
  consul:
    client:
      address: "xmidt-consul-server.rdk-shared-uat:8500"
      scheme: "http"
      waitTime: "30s"
    disableGenerateID: true
    vnodeCount: 211
    watches:
      -
        service: "talaria"
        tags:
          - "env=dev"
        passingOnly: true
    registrations:
    -
      id: "talaria-dev-0"
      name: "talaria"
      tags:
        - "env=dev"
      address: "http://xmidt-talaria-0"
      scheme: "http"
      port: 6200
      checks:
        -
          checkID: "talaria-dev-0:http"
          http: "http://xmidt-talaria-0.rdk-shared-dev:6201/health"
          interval: "10s"
          deregisterCriticalServiceAfter: "45s"
tracing:
  provider: "stdout"
  skipTraceExport: true

Please let me know if you need anything else from my side.

cc: @schmidtw

[johnabass]

Do you have any metrics from the affected talarias? It would be helpful to see prometheus results just before and just after the rehash events that you're seeing.

[Sachin4403]

Here the metrics for all talaria's

rehasher.pdf

[JC000]

In your metrics, can you capture the histogram (graph) instead of current value?

[Sachin4403]

Hi @JC000, Here are graph details from grafana.

Untitled.1.mp4

[johnabass]

There are docker files in each of our repos that you can use to verify where the problem might be. The talaria Dockerfile will run a talaria that you can configure locally. You will probably have to tweak your configuration a bit, as your consul setup might not work exactly the same.

[Sachin4403]

Will try to use this and let you know the results.

[Sachin4403]

Hi @johnabass,
I had tried the same using local docker-compose and the results are the same. Attaching a video of results.

local_setup.mp4

[JC000]

I don't know if your description of your configuration is accurate, or if you have actually mis-configured Kratos. If your client is configured to connect directly to a Talaria instance (instead of a Petasos instance), then that client doesn't know how to connect to other instances. It is stuck to that one Talaria instance.

In addition, if your cluster configuration changes from 1 Talaria nodes to 4 Talaria nodes, you should expect a great deal of flapping. The cluster make up as reflected in Consul isn't atomic. The change as seen by a single Talaria may be "1 -> 3 -> 4", or "1 -> 2 -> 4" or some other combination. The algorithm allows for eventual consistency, but it does not prescribe the path to that consistency unless the delta is exactly 1.

[Sachin4403]

Right @JC000, Petasos is just routing to any of the talaria and store it in memory for further use. but when the consul is in place scytale won't connect to petasos as it will get all the talaria's from the consul and fanout to all of them.

I just wanted to check about the load testing on talaria and rehashing across talaria instances, that's why I had just configured one talaria in Kratos.

[JC000]

My comment on the use of Petasos is not about how Scytale connects, it's about how clients connects. Any client (Parodus or Kratos) configured to connect directly to a Talaria can never be "rehashed". It can only be disconnected.

If you wish to test "rehashing across Talaria instances", your clients must be configured to connect to your Petasos, regardless of how many Talaria instances you have deployed. Configuring a client to connect directly to a Talaria works only in the simple case where only one Talaria instance is being used.

[Sachin4403]

Hi @JC000, even after trying with petasos with simulator devices the results are the same. you can check them in the local_setup.mp4 file.

[JC000]

At this point, I think we'll need to look at logs from the simulators. I'm not convinced they're configured correctly.

[Sachin4403]

Hi @JC000, I am using the default configuration of docker-compose, only petasos is modified as there was no service discovery.

[Sachin4403]

Hi @JC000 @johnabass
Any update on this?

[JC000]

As I mentioned on July 1st, we'll need logs. We cannot reproduce your problem.

[Sachin4403]

Hi @JC000 and @johnabass

Here are the logs for 6 simulators and Kratos.
Logs of 6 Simulators and kratos.txt
let me know if something else is needed from my end.

[JC000]

Can you also send the Talaria logs?

[Sachin4403]

Hi @JC000, Here are the talaria logs.

talaria0 logs.txt

[johnabass]

I just noticed your consul section may not be identifying each talaria appropriately:

     registrations:
    -
      id: "talaria-dev-0"  <--- this needs to be unique for each talaria in the cluster
      name: "talaria"
      tags:
        - "env=dev"
      address: "http://xmidt-talaria-0"  <--- this should normally just be xmidt-talaria-0
      scheme: "http"
      port: 6200

Are you configuring each talaria to register with a unique service id? If not, then consul thinks there is only one talaria. Bringing up a new one with the same id will replace the old one, causing a rehash. That's expected behavior, as it's what would happen if you were hotswapping a talaria instance.

Since our deployments have each talaria using a different FQDN, we use the FQDN as our service id. But it doesn't matter what you use, as long as it's unique for each talaria instance.

Also, make sure services are registered with different addresses. More than one service with the same address is allowed by consul, but our application software throws away duplicates for the purposes of hashing.

[Sachin4403]

Hi @johnabass,
id and address will be unique always.

here is the sample for the same

For talaria0

registrations:
        -
          id: "talaria-dev-0"
          name: "talaria"
          tags:
            - "env=dev"
          address: "http://xmidt-talaria-0"
          enableTagOverride: true
          scheme: "http"
          port: 6200
          checks:
            -
              checkID: "talaria-dev-0:http"
              http: "http://xmidt-talaria-0.6201/health"
              interval: "10s"
              deregisterCriticalServiceAfter: "45

For talaria1

 registrations:
        -
          id: "talaria-dev-1"
          name: "talaria"
          tags:
            - "env=dev"
          address: "http://xmidt-talaria-1"
          enableTagOverride: true
          scheme: "http"
          port: 6200
          checks:
            -
              checkID: "talaria-dev-1:http"
              http: "http://xmidt-talaria-1:6201/health"
              interval: "10s"
              deregisterCriticalServiceAfter: "45s"
      

[JC000]

@Sachin4403 : After looking through the log, the issue that @johnabass mentioned is your problem. He pointed out the address attribute should be a hostname, not an URL. This is also shown in the logs:

{"action":"register","address":"http://c982b1ac2a7e","id":"c982b1ac2a7e","instance":"http://http://c982b1ac2a7e:6200","service":"talaria","tags":"[stage=dev flavor=mint]","ts":"2021-07-22T08:12:36.5143628Z"}

Note that the incorrect use of an URL caused the instance to have http:// repeated.

[Sachin4403]

Hi @JC000 and @johnabass the log files are from the local docker compose set which was created using the xmidt repo. so this double http issue has to be fixed there and the log line with double http

{"action":"register","address":"http://c982b1ac2a7e","id":"c982b1ac2a7e","instance":"http://http://c982b1ac2a7e:6200","service":"talaria","tags":"[stage=dev flavor=mint]","ts":"2021-07-22T08:12:36.5143628Z"}

will be printed when a talaria will be self registering himself in consul. here instance and id will be injected to logger and will be printer in logs from here.

when talaria is trying to get instance from consul and it is able to get with proper address.here is the example for the same

{"datacenter":"","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-22T08:12:36.5151984Z"}
{"datacenter":"","eventCount":1,"instances":[],"level":"error","msg":"service discovery update","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-22T08:12:36.5152378Z"}
{"datacenter":"","eventCount":1,"level":"info","msg":"ignoring initial instances","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-22T08:12:36.5152988Z"}
{"datacenter":"","eventCount":2,"instances":["http://847f629274bc:6200"],"level":"error","msg":"service discovery update","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-22T08:12:44.8076163Z"}

Let me know if anything else is needed from my end.

[JC000]

It is possible that the docker compose files in the Xmidt repo contains bugs. They have only been verified to work as an all-in-one type configuration for testing purposes. If you can provide a PR for a fix, we would be happy to merge it.

[Sachin4403]

Hi @JC000
I had raised the PR to fix the issue in the docker-compose set up xmidt-org/talaria#190.

[Sachin4403]

Hi @JC000 @johnabass,

Thanks for your inputs, it was a configuration issue which I had corrected it post that rehasher is working fine as it is expected.

cc: @schmidtw