Kong DP node killed with SIGQUIT during startup with large configuration #11651
Comments
|
CPU and Memory of CP nodes are too high during the start
|
Something sent the DP SIGQUIT. |
|
@hbagdi thanks for the response. Yes Official docs for CP/DP doesnt talk about how the deployment model should look like (Should DP have persistent volumes attached or not etc), how the initial DP boot up happens in case of large amount of data present in CP |
It's usually sent by command like
See here hybrid mode overview. How many entities do you have? If there are too many, it could significantly impact your CPU usage and MEMORY usage. |
|
@chobits Thanks for the response.
Went through this, this doc does not talk about how to persist DP config across restarts ( like in kubernetes environments ), Also does not talk more about why TTL in schema does not work in plugins, what we need to consider while developing custom plugins in CP/DP model etc are missing.
Number of entities are large, Like we have 1M consumers, 2M Basic auth credentials etc. CP alway tries to send the whole config in one shot, there is no config to batch this |
This is a bug of old version. And has been fixed in the latest master branch. See here: #11464
Yeah, it's a known performance issue for the current kong version. We're working on addressing it and plan to But having a million entities is excessive, you might want to consider reducing them from a business standpoint. |
|
Thanks for detailed response. Will wait for the performance fixes
But having millions of users is not abnormal right, there should be a way to support such usecases IMO |
ok. keep a glance to the update of kong project:) |
|
Could it be that the DP does not turn healthy and thus is killed, with SIGQUIT, by its supervisor (I.e. docker-compose, Kubernetes). Can you increase the timeout that is allotted to DP processes? Processing the initial configuration (and subsequent updates) will take a lot of time if you have millions of users, and there currently is no way to avoid that in hybrid mode. |
hanshuebner
added
the
pending author feedback
|
Thanks @hanshuebner for the response. Yes i have tried that, dataplane will stay in pending state even after 2 mins, And have seen control plane also gets crashed due to high node memory usage. May be for our use case CP/DP might not be ready as of now without batched updates to DP nodes |
|
At this point, it is safe to say that millions of consumers are not supported in hybrid mode. We don't have a specific maximum number of consumers that we can specify. The whole configuration is sent from the control plane to the data plane, and each consumer is an object in that configuration. The resulting configuration file will simply be too large if you have millions of consumers. Note that full configurations are also sent when the configuration is updated, so the excessive memory usage will occur not only during startup, but with each configuration change. We're currently discussing both incremental configuration updates in hybrid mode and other ways how we can improve Kong's authentication system. Unfortunately, we cannot give you a firm date as to when this will become reality. |
hanshuebner
changed the title
|
I'm closing this issue even though we could not provide you with a solution at this point. We are aware of the limitation and hope to be able to solve the issue in a future release. |
Is there an existing issue for this?
Kong version (
$ kong version)3.4.0
Current Behavior
During the startup DP nodes crash without any error in the logs
CP Logs
DP Logs
Expected Behavior
DP Node should come up properly
Steps To Reproduce
Anything else?
Trying to setup CP/DP setup for the first time with the existing Kong deployment, existing deployment already has lot of data (Increased
cluster_max_payloadvalue for the same). How it is usually done ? All the data from CP to DP is sent in a single shot ? Is there anyway to send this in batches ?The text was updated successfully, but these errors were encountered: