Shutting down context by removing known leaks freezes application

[bvajk]

I am using the servlet 2.5 version with the listener correctly before any other, and in an attempt to solve this I also added
ClassLoaderLeakPreventor.executeShutdownHooks - false
On shutdown I get this as the last log entries before the freeze

2024-02-09 13:58:49,447 WARN  [org.springframework.beans.factory.support.DisposableBeanAdapter] (ServerService Thread Pool -- 218) Destroy method 'destroy' on bean with name 'grapevine.BackgroundRunner' threw an exception: java.lang.IllegalMonitorStateException
2024-02-09 13:58:49,806 INFO  [stdout] (ServerService Thread Pool -- 218) ClassLoaderLeakPreventorListener: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener shutting down context by removing known leaks (CL: 0x4f041f13)
2024-02-09 13:58:49,839 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) private java.util.Map javax.faces.component.UIComponentBase.descriptors is not static
2024-02-09 13:58:49,850 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,850 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,850 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,851 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,852 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,852 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting
2024-02-09 13:58:49,852 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 218) AccessControlContext of javax.imageio.spi.IIORegistry used custom combiner - unsetting

How can I go about solving this issue?

[mjiderhamn]

Are you able to get a heap dump in the frozen state?

[bvajk]

I've tried getting a heap dump with jcmd, but I wasn't able to and got this error message:

com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
        at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
        at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
        at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147)
        at sun.tools.jcmd.JCmd.main(JCmd.java:131)

But I did manage to get a threaddump, in which I found a single thread related to the library and it looked like this:

Thread 10592: (state = IN_NATIVE)
 - sun.nio.fs.UnixNativeDispatcher.access0(long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.fs.UnixNativeDispatcher.access(sun.nio.fs.UnixPath, int) @bci=10, line=454 (Compiled frame)
 - sun.nio.fs.UnixFileSystemProvider.checkAccess(java.nio.file.Path, java.nio.file.AccessMode[]) @bci=202, line=306 (Compiled frame)
 - java.nio.file.Files.isAccessible(java.nio.file.Path, java.nio.file.AccessMode[]) @bci=6, line=2455 (Compiled frame)
 - java.nio.file.Files.isReadable(java.nio.file.Path) @bci=11, line=2490 (Compiled frame)
 - org.jboss.as.logging.LoggingResource$1.visitFile(java.nio.file.Path, java.nio.file.attribute.BasicFileAttributes) @bci=53, line=269 (Compiled frame)
 - org.jboss.as.logging.LoggingResource$1.visitFile(java.lang.Object, java.nio.file.attribute.BasicFileAttributes) @bci=6, line=252 (Compiled frame)
 - java.nio.file.Files.walkFileTree(java.nio.file.Path, java.util.Set, int, java.nio.file.FileVisitor) @bci=105, line=2670 (Compiled frame)
 - org.jboss.as.logging.LoggingResource.findFiles(java.lang.String, org.jboss.dmr.ModelNode, boolean) @bci=55, line=252 (Interpreted frame)
 - org.jboss.as.logging.LoggingResource.getChildrenNames(java.lang.String) @bci=37, line=158 (Interpreted frame)
 - org.jboss.as.logging.LoggingResource.hasReadableFile(java.lang.String) @bci=3, line=230 (Compiled frame)
 - org.jboss.as.logging.LoggingResource.getChild(org.jboss.as.controller.PathElement) @bci=17, line=108 (Compiled frame)
 - org.jboss.as.controller.OperationContextImpl.getAuthorizationResource(org.jboss.as.controller.PathAddress) @bci=113, line=1788 (Compiled frame)
 - org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(org.jboss.as.controller.AbstractOperationContext$OperationId, org.jboss.dmr.ModelNode) @bci=55, line=1745 (Compiled frame)
 - org.jboss.as.controller.OperationContextImpl.authorizeResource(boolean, boolean) @bci=130, line=1282 (Compiled frame)
 - org.jboss.as.controller.OperationContextImpl.authorizeResource(boolean, boolean) @bci=3, line=133 (Compiled frame)
 - org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$CheckResourceAccessHandler.execute(org.jboss.as.controller.OperationContext, org.jboss.dmr.ModelNode) @bci=28, line=429 (Compiled frame)
 - org.jboss.as.controller.AbstractOperationContext.executeStep(org.jboss.as.controller.AbstractOperationContext$Step) @bci=33, line=890 (Compiled frame)
 - org.jboss.as.controller.AbstractOperationContext.processStages() @bci=158, line=659 (Compiled frame)
 - org.jboss.as.controller.AbstractOperationContext.executeOperation() @bci=27, line=370 (Compiled frame)
 - org.jboss.as.controller.OperationContextImpl.executeOperation() @bci=20, line=1349 (Compiled frame)
 - org.jboss.as.controller.ModelControllerImpl.internalExecute(org.jboss.dmr.ModelNode, org.jboss.as.controller.client.OperationMessageHandler, org.jboss.as.controller.ModelController$OperationTransactionControl, org.jboss.as.controller.client.OperationAttachments, org.jboss.as.controller.OperationStepHandler, boolean, boolean) @bci=673, line=395 (Compiled frame)
 - org.jboss.as.controller.ModelControllerImpl.execute(org.jboss.dmr.ModelNode, org.jboss.as.controller.client.OperationMessageHandler, org.jboss.as.controller.ModelController$OperationTransactionControl, org.jboss.as.controller.client.OperationAttachments) @bci=12, line=207 (Compiled frame)
 - org.jboss.as.jmx.model.ResourceAccessControlUtil.getResourceAccess(org.jboss.as.controller.PathAddress, boolean) @bci=59, line=85 (Interpreted frame)
 - org.jboss.as.jmx.model.RootResourceIterator.doIterate(org.jboss.as.controller.registry.Resource, org.jboss.as.controller.PathAddress) @bci=25, line=51 (Compiled frame)
 - org.jboss.as.jmx.model.RootResourceIterator.doIterate(org.jboss.as.controller.registry.Resource, org.jboss.as.controller.PathAddress) @bci=162, line=61 (Interpreted frame)
 - org.jboss.as.jmx.model.RootResourceIterator.doIterate(org.jboss.as.controller.registry.Resource, org.jboss.as.controller.PathAddress) @bci=162, line=61 (Interpreted frame)
 - org.jboss.as.jmx.model.RootResourceIterator.iterate() @bci=8, line=43 (Interpreted frame)
 - org.jboss.as.jmx.model.ModelControllerMBeanHelper.queryNames(javax.management.MBeanServer, javax.management.ObjectName, javax.management.QueryExp) @bci=28, line=195 (Interpreted frame)
 - org.jboss.as.jmx.model.ModelControllerMBeanServerPlugin.queryNames(javax.management.ObjectName, javax.management.QueryExp) @bci=52, line=216 (Interpreted frame)
 - org.jboss.as.jmx.PluggableMBeanServerImpl.queryNames(javax.management.ObjectName, javax.management.QueryExp) @bci=107, line=838 (Interpreted frame)
 - se.jiderhamn.classloader.leak.prevention.cleanup.MBeanCleanUp.cleanUp(se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor) @bci=15, line=25 (Interpreted frame)
 - se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor.runCleanUps() @bci=128, line=246 (Interpreted frame)
 - se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventorListener.contextDestroyed(javax.servlet.ServletContextEvent) @bci=107, line=242 (Interpreted frame)
 - io.undertow.servlet.core.ApplicationListeners.contextDestroyed() @bci=48, line=202 (Interpreted frame)
 - io.undertow.servlet.core.DeploymentImpl.destroy() @bci=4, line=220 (Interpreted frame)
 - io.undertow.servlet.core.DeploymentManagerImpl.undeploy() @bci=16, line=598 (Interpreted frame)
 - org.wildfly.extension.undertow.deployment.UndertowDeploymentService.stopContext() @bci=94, line=145 (Interpreted frame)
 - org.wildfly.extension.undertow.deployment.UndertowDeploymentService$2.run() @bci=4, line=121 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1149 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
 - org.jboss.threads.JBossThread.run() @bci=10, line=320 (Interpreted frame)

[bvajk]

Interestingly on my local tomcat server this problem does not occur. The next log entry (which is supposed to be following the freeze) is 13-Feb-2024 10:04:16.124 WARNING [RMI TCP Connection(12)-127.0.0.1] se.jiderhamn.classloader.leak.prevention.JULLogger.warn JDBC driver loaded by protected ClassLoader deregistered: class org.postgresql.Driver
And org.postgres is provided by jboss in our remote configuration.
Could the issue be that the library is trying to deregister a class which is provided by the Jboss EAP?

[bvajk]

I've made org.postgres not be provided in our jboss build config and I can now see
2024-02-13 13:57:25,341 WARNING [se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor] (ServerService Thread Pool -- 258) JDBC driver loaded by protected ClassLoader deregistered: class org.postgresql.Driver in jboss logs
This gives me another reason to think provided dependencies might be the cause of the issue

[bvajk]

I haven't been able to further investigate this.
My current best guess is that somehow the unsetting process gets stuck on either the provided dependencies or JNDI resources.
I'll try to add this library again when we migrate to tomcat instances on production