Nashorn JavaScript engine used for adaptive authentication causes memory leak with JDK 11

[tharakawijekoon]

Describe the issue:
Nashorn JavaScript engine used for adaptive authentication has been removed/deprecated in newer JDK versions. Also, there are bugs in JDK 11 which cause memory leaks when using the Nashorn JavaScript engine. These JDK bugs are not fixed[1]

How to reproduce OOM issue due to JDK bug[1]:

• Configure a Service provider with the following adaptive authentication script.

var onLoginRequest = function onLoginRequest(context) {
  executeStep(1,
      {
        onSuccess: function (context) {
          var fedUser = context.currentKnownSubject;
          var currentIdp = context.currentStep.idp;
          var endUser = fedUser;          
          fedUser = null;
          currentIdp = null;
          endUser = null;
        }});
};

• Login/SSO/logout to the service provider repeatedly with high load. The server would go OOM.

Expected behavior:
If the JDK bug[1] is not fixed and JDK does not support it, look for alternatives for the Nashorn JavaScript engine.

Environment information :

• Product Version: [IS-5.11.0]

[1]https://bugs.openjdk.java.net/browse/JDK-8229011

[ruwanta]

duplicate #6404

[ruwanta]

I think this has three concerns mixed.

1. Nashorn is not supported on JDK : Here Nashorn has been moved out from JDK to its own library and is maintained under LGPL. Hence there is no question on this regard.
2. Bug 8229011 , I do not see that is a memory leak. It creates lots of engines inside the for loop. That supposed to consume memory indefinitely and should cause OOM. Try the same with any other object which has a lifecycle, you will notice this. e.g. try reading a web page with new HTTP client inside an infinite loop.
3. The code block stated here also related to above (2). Engine is created per each request (for context isolation) if you execute very high number of requests. I do not see this is memory leak, as long as we allocate enough memory for the purpose and it does not re-claim after the high number of requests are reduced.

[shanggeeth]

An alternative script execution engine in introduced with wso2/carbon-identity-framework#5279.