ARSN-370: fix memory leak in MongoDBReadStreams #2177

williamlardier · 2023-10-05T09:14:09Z

The MongoDBReadStreams are not properly destroyed in both the Bucket V1 and V0 cases. In the V1 case, only the pipe-ed stream, the Transform one, is cleaned. In the V0 case, we directly call the callback without properly cleaning the stream, leaving open, in both cases, the mongodb cursors, that in turn affect the mongos memory consumption.

The fix was tested end2end:

Before (with k8s memory limits, otherwise it uses up to 100% of the cluster RAM):

Causing heap overflows in cloudserver pods and high MongoS memory usage due to open cursors.

After:

bert-e · 2023-10-05T09:14:12Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2023-10-05T09:14:20Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

codecov-commenter · 2023-10-05T10:16:24Z

Codecov Report

Merging #2177 (17b5bbc) into development/8.1 (af34571) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@                 Coverage Diff                 @@
##           development/8.1    #2177      +/-   ##
===================================================
+ Coverage            64.28%   64.34%   +0.05%     
===================================================
  Files                  210      210              
  Lines                16381    16391      +10     
  Branches              3320     3320              
===================================================
+ Hits                 10531    10547      +16     
+ Misses                5835     5829       -6     
  Partials                15       15

Files	Coverage Δ
...orage/metadata/mongoclient/MongoClientInterface.js	`63.18% <100.00%> (+0.73%)`	⬆️

... and 1 file with indirect coverage changes

francoisferrand · 2023-10-05T12:09:50Z

lib/storage/metadata/mongoclient/MongoClientInterface.js

@@ -1736,10 +1743,14 @@ class MongoClientInterface {
                };
                log.error(
                    'internalListObjectV1: error listing objects', logObj);
+                // call explicitly the destroy method to clean the mongodb cursor
+                stream.destroy();


is this call not duplicated: e.g. will end not be called on error as well?

No, and actually it's not even called in the v1 case due to the piping. See 4aa8b5c

if I remove the line, the test would fail because the stream was not properly cleaned.

francoisferrand · 2023-10-05T12:10:16Z

tests/functional/metadata/mongodb/listObject.spec.js

@@ -477,6 +479,25 @@ describe('MongoClientInterface::metadata.listObject', () => {
                    }),
                ], done);
            });
+


can we add a test for the "error" case as well?

Done in 4aa8b5c

bert-e · 2023-10-06T07:07:34Z

Incorrect fix version

The Fix Version/s in issue ARSN-370 contains:

8.1.112

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

8.1.113

Please check the Fix Version/s of ARSN-370, or the target
branch of this pull request.

The MongoDBReadStreams are not properly destroyed in both the Bucket V1 and V0 cases. In the V1 case, only the pipe-ed stream, the Transform one, is cleaned. In the V0 case, we directly call the callback without properly cleaning the stream, leaving open, in both cases, the mongodb cursors, that in turn affect the mongos memory consumption.

williamlardier · 2023-10-06T11:50:49Z

/approve

bert-e · 2023-10-06T11:51:00Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/8.1

The following branches will NOT be impacted:

development/6.4
development/7.10
development/7.4
development/7.70

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2023-10-06T11:51:10Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/8.1

The following branches have NOT changed:

development/6.4
development/7.10
development/7.4
development/7.70

Please check the status of the associated issue ARSN-370.

Goodbye williamlardier.

williamlardier force-pushed the bugfix/ARSN-370-fix-listing-memory-leak branch 3 times, most recently from b7226d3 to 6aaace5 Compare October 5, 2023 10:09

williamlardier force-pushed the bugfix/ARSN-370-fix-listing-memory-leak branch from 6aaace5 to aabbdc1 Compare October 5, 2023 10:20

williamlardier requested review from nicolas2bert, francoisferrand, KillianG, Kerkesni and benzekrimaha October 5, 2023 10:21

francoisferrand reviewed Oct 5, 2023

View reviewed changes

francoisferrand approved these changes Oct 5, 2023

View reviewed changes

nicolas2bert approved these changes Oct 5, 2023

View reviewed changes

benzekrimaha approved these changes Oct 5, 2023

View reviewed changes

williamlardier force-pushed the bugfix/ARSN-370-fix-listing-memory-leak branch from aabbdc1 to 1cb6cb8 Compare October 6, 2023 07:07

williamlardier added 3 commits October 6, 2023 09:13

ARSN-370: handle error cases

4aa8b5c

ARSN-370: bump project version

17b5bbc

williamlardier force-pushed the bugfix/ARSN-370-fix-listing-memory-leak branch from 1cb6cb8 to 17b5bbc Compare October 6, 2023 07:14

bert-e merged commit 17b5bbc into development/8.1 Oct 6, 2023
7 checks passed

bert-e deleted the bugfix/ARSN-370-fix-listing-memory-leak branch October 6, 2023 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARSN-370: fix memory leak in MongoDBReadStreams #2177

ARSN-370: fix memory leak in MongoDBReadStreams #2177

williamlardier commented Oct 5, 2023 •

edited

Loading

bert-e commented Oct 5, 2023

bert-e commented Oct 5, 2023

codecov-commenter commented Oct 5, 2023 •

edited

Loading

francoisferrand Oct 5, 2023

williamlardier Oct 6, 2023

francoisferrand Oct 5, 2023

williamlardier Oct 6, 2023

bert-e commented Oct 6, 2023

williamlardier commented Oct 6, 2023

bert-e commented Oct 6, 2023

bert-e commented Oct 6, 2023

ARSN-370: fix memory leak in MongoDBReadStreams #2177

ARSN-370: fix memory leak in MongoDBReadStreams #2177

Conversation

williamlardier commented Oct 5, 2023 • edited Loading

bert-e commented Oct 5, 2023

Hello williamlardier,

bert-e commented Oct 5, 2023

Waiting for approval

codecov-commenter commented Oct 5, 2023 • edited Loading

Codecov Report

francoisferrand Oct 5, 2023

Choose a reason for hiding this comment

williamlardier Oct 6, 2023

Choose a reason for hiding this comment

francoisferrand Oct 5, 2023

Choose a reason for hiding this comment

williamlardier Oct 6, 2023

Choose a reason for hiding this comment

bert-e commented Oct 6, 2023

Incorrect fix version

williamlardier commented Oct 6, 2023

bert-e commented Oct 6, 2023

In the queue

bert-e commented Oct 6, 2023

williamlardier commented Oct 5, 2023 •

edited

Loading

codecov-commenter commented Oct 5, 2023 •

edited

Loading