High Memory Consumption in ETCD During Frequent Updates

[pratiks-alepo]

Hello Team,

I am writing to bring your attention to an issue we have been experiencing with ETCD, specifically related to high memory consumption when frequent updates are performed.

We are currently using version 3.4.15 of ETCD, and we have also tested the latest version available, but the behavior remains the same. Whenever we perform frequent updates in ETCD, we notice a significant increase in memory usage, which eventually leads to performance degradation and potential out-of-memory errors.

We have attempted to mitigate this issue by adjusting the snapshot settings as well, but unfortunately, the problem persists. Despite these efforts, the memory consumption continues to rise steadily as the number of updates increases.

We kindly request your assistance in addressing this issue. We would greatly appreciate any guidance, recommendations, or potential solutions that could help alleviate the high memory consumption problem during frequent updates.

Additionally, if there are any known fixes or updates available specifically targeting this memory issue in ETCD, please let us know. We are eager to explore all available options to ensure optimal performance and stability in our environment.

Thank you very much for your attention to this matter. We look forward to your prompt response and any assistance you can provide in resolving this issue.

Best regards,
Pratik

[jmhbnz]

Hi @pratiks-alepo - Thanks for raising this discussion, can you please provide a detailed example of what you mean by "frequent updates"? I.e. what is the operation being performed, including the payload sizes and at what scale?

Additionally can you please provide the configuration file or parameters that you are using for etcd currently?

It would also be helpful context if you could provide some metrics or a summary of what is happening in terms of memory usage.

[pratiks-alepo]

Hi @fuweid ,

Thanks for your response.
We are in process to investigate the behavior as mentioned by @chaochn47 and will provide the observed results and any relevant finding as soon as possible.

Best Regards,
Pratik

[pratiks-alepo]

Hello @chaochn47 @fuweid ,

We have completed our analysis regarding the ongoing issue related to ETCD memory growth.Here are our observations:

1. ETCD Key Update: We are updating a single key(as mentioned in earlier post) in ETCD at regular intervals of 10 seconds.

2. Memory Growth: As a result of the frequent key updates, we have observed a steady increase in memory consumption.

3. Database Size: The size of the database is also growing.

To provide more insight into the situation, we have attached the ETCD memory graph and a table illustrating the correlation between the database size and memory consumption.

We welcome any suggestions or insights from the team regarding this matter.

<style> </style>

Time	ETCD DB-SIZE (in MB)	ETCD Pod Memory (in MB)
06:12:25 UTC 2023	8.6	42
06:17:25 UTC 2023	11	44
07:27:26 UTC 2023	47	51
08:02:26 UTC 2023	66	53
08:32:27 UTC 2023	81	58
09:02:28 UTC 2023	96	62
09:32:28 UTC 2023	112	66
10:02:29 UTC 2023	127	68
10:32:30 UTC 2023	141	76

Best Regards,
Pratik

[fuweid]

Hi @pratiks-alepo according to your last reply, the key overload-control/service1/service/status has history versions which seems reasonable to me. You can checkout the https://etcd.io/docs/v3.5/op-guide/maintenance/ to set the auto-compaction-mode to delete the history. Hope it can help.

[pratiks-alepo]

Hi @fuweid ,

Thanks for your response. We will go through the link and try out some options.
We will get back with our results on this.

Best Regards,
Pratik

[pratiks-alepo]

Hello @fuweid,

Thank you for sharing the link. We made some changes to meet our requirements and noticed significant improvement. Here is the current configuration:

• ETCD_AUTO_COMPACTION_MODE: revision
• ETCD_AUTO_COMPACTION_RETENTION: 2
• ETCD_SNAPSHOT_COUNT: 500
• ETCD_MAX_SNAPSHOTS: 2
• ETCD_MAX_WALS: 4

Despite these changes, we are still experiencing a memory leak. Over a span of 7 days, we observed a leak of approximately 30MB, with 20,373,662 update operations performed on the same key. The graph below illustrates this trend.

We would appreciate your assistance in identifying any additional changes we can make to address this memory leak issue.

Best regards,
Pratik