Dynalink leaks memory when adaptive authentication scripts are used with JDK 11

[deshankoswatte]

Describe the issue:

When adaptive authentication scripts are used with JDK 11 a memory leak identified at JDK's end [1] causes the WSO2 Identity Server (WSO2 IS) to go out of memory (OOM). The heap memory rises and it does not get cleared up.

How to reproduce:

• Create a service provider and add a simple adaptive authentication script. In my case, I called a function that iterated through a for loop before executing the first step.
• Create a JMeter script to call the authorize endpoint infinitely. I configured the thread count and ramp-up to be 10.
• Execute the JMeter script till it causes the WSO2 IS to run OOM.
• Extract a heap dump and analyze the heap dump (by retaining unreachable objects), leaks similar to the one mentioned at [1] will be noted.

In my case, the -Xmx was configured to the default value of 1024, and the authorize calls sent during that time added up to about 1 million requests.

Sample screenshots with the evidence are attached below:

Heap Dump Analysis:

JConsole Overview during the test:

Expected behavior:

The fix for the issue is provided with JDK 17 but it has not been sent out to JDK 11. If the fix is not ported to JDK 11 alternatives should be looked into.

Environment information (Please complete the following information; remove any unnecessary fields) :

• Product Version: IS 5.10.0.243
• OS: Ubuntu 22.04.2 LTS
• Database: H2
• Userstore: LDAP

References:

[1] - https://bugs.openjdk.org/browse/JDK-8198540

---

[deshankoswatte]

The aforementioned issue was tested with WSO2 IS 6.0.0.119 using JDK 17.0.7, and it appears that the issue has been resolved. The same test mentioned above was conducted for approximately 3 hours, and the JConsole results were as follows. As you can observe, the heap now clears properly when compared to the screenshot included in the previous comment.

Furthermore, we acquired a heap dump and analyzed it to verify the presence of Dynalink objects. However, we could not locate any such objects in the two heap dumps that were obtained. The screenshot below provides further illustration.

[hwupathum]

The above issue was tested with WSO2 IS 5.10.0.248 with Java 11.0.20. I used a JMeter script to call the authorize endpoint indefinitely and was able to observe the same increase of memory. I observed that the Old Gen memory is increasing linearly with the number of executions and becomes static when the test script is stopped.

Old Generation Heap:

Total Heap:

Tested the same script in Java 8 and noticed that the Old Generation memory is reduced frequantly than with Java 11.

Old Generation Heap:

Total Heap:

[shanggeeth]

Temporary fix: wso2/carbon-identity-framework#5202

[shanggeeth]

An alternative script execution engine in introduced with wso2/carbon-identity-framework#5279.