Processing Pattern should use non-Coherence threads #145

brianoliver · 2015-08-27T14:00:24Z

It appears that Processing Pattern uses Coherence (service) threads. If so please consider using non-Coherence threads.

brianoliver · 2015-08-27T14:00:24Z

Reported by janneman

brianoliver · 2016-05-23T16:12:46Z

@brianoliver said:
An example stack trace:

2016-05-12 16:01:41
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.72-b04 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007ffa88002000 nid=0x5f5d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"DefaultTaskProcessor-TPSK:{Identifier

{GridTaskProcessor},1,0x0AF194B700000154A5046D2D181A69EB}:Thread-618" daemon prio=10 tid=0x00007ff8e0008800 nid=0x4acb waiting for monitor entry [0x00007ff99f5f4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession.acquireSubmission(DefaultProcessingSession.java:483)

waiting to lock <0x0000000717d00068> (a java.lang.Object)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:104)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:131)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getEmptySubmissions(ActivityGridServiceImpl.java:146)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityTaskLoaderControl.calculateHeadRoom(ActivityTaskLoaderControl.java:94)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityTaskLoaderControl.getBatchOfTaskIDs(ActivityTaskLoaderControl.java:68)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityLoaderSession.loadNextBatch(ActivityLoaderSession.java:109)
at com.foo.hi.activityprocessing.domain.internal.ActivityLoaderTask.run(ActivityLoaderTask.java:83)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier{GridTaskProcessor}

,1,0x0AF194B700000154A5046D2D181A69EB}:Thread-617" daemon prio=10 tid=0x00007ff8e0007800 nid=0x4aca waiting for monitor entry [0x00007ff8bf11b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession.acquireSubmission(DefaultProcessingSession.java:485)

waiting to lock <0x0000000717d00068> (a java.lang.Object)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:104)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:131)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.isSubmissionCompleted(BaseResumableTask.java:134)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.finish(BaseResumableTask.java:117)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.handleCompletion(BaseResumableTask.java:102)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.handleResult(BaseResumableTask.java:74)
at com.foo.hi.activityprocessing.domain.internal.ActivityLoaderTask.run(ActivityLoaderTask.java:90)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier

{GridTaskProcessor},1,0x0AF194B700000154A5046D2D181A69EB}:Thread-616" daemon prio=10 tid=0x00007ff8e0007000 nid=0x4ac9 waiting on condition [0x00007ff8bf019000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflowComplex(Service.CDB:68)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflow(Service.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.drainOverflow(Grid.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.invoke(PartitionedCache.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.invoke(PartitionedCache.CDB:36)
at com.tangosol.coherence.component.util.SafeNamedCache.invoke$Router(SafeNamedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.invoke(SafeNamedCache.CDB:5)
at com.foo.coherence.common.util.NamedCacheObjectProxy.invoke(NamedCacheObjectProxy.java:111)
at com.sun.proxy.$Proxy286.getSubmissionState(Unknown Source)
at com.foo.coherence.patterns.processing.internal.DefaultSubmissionOutcome.checkSubmissionResult(DefaultSubmissionOutcome.java:294)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession.acquireSubmission(DefaultProcessingSession.java:501)

locked <0x0000000717d00068> (a java.lang.Object)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:104)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:131)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.isSubmissionCompleted(BaseResumableTask.java:134)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.finish(BaseResumableTask.java:117)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.handleCompletion(BaseResumableTask.java:102)
at com.foo.hi.support.gridprocessing.task.BaseResumableTask.handleResult(BaseResumableTask.java:74)
at com.foo.hi.support.gridprocessing.task.BaseActivityTask.run(BaseActivityTask.java:87)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier{GridTaskProcessor}

,1,0x0AF194B700000154A5046D2D181A69EB}:Thread-609" daemon prio=10 tid=0x00007ff8e0006000 nid=0x4aba waiting for monitor entry [0x00007ffa461e7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession.submit(DefaultProcessingSession.java:434)

waiting to lock <0x0000000717d00068> (a java.lang.Object)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.submit(ActivityGridServiceImpl.java:298)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.submitAndRelease(ActivityGridServiceImpl.java:52)
at com.foo.hi.activityprocessing.domain.internal.components.StreamingAccumulatedSpawnStrategy.completeGroup(StreamingAccumulatedSpawnStrategy.java:33)
at com.foo.hi.activityprocessing.domain.internal.components.BaseAccumulatedSpawnStrategy.spawn(BaseAccumulatedSpawnStrategy.java:42)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityTaskSpawner.spawnTasks(ActivityTaskSpawner.java:61)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityLoaderSession.spawn(ActivityLoaderSession.java:122)
at com.foo.hi.activityprocessing.domain.internal.ActivityLoaderTask.run(ActivityLoaderTask.java:84)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier

{GridTaskProcessor},1,0x0AF194B700000154A5046D2D181A69EB}:Thread-608" daemon prio=10 tid=0x00007ff8e0005000 nid=0x4ab8 waiting on condition [0x00007ff8bf21c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflowComplex(Service.CDB:68)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.drainOverflow(Service.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.drainOverflow(Grid.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.poll(Grid.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.containsKey(PartitionedCache.CDB:24)
at com.foo.common.collections.ConverterCollections$ConverterMap.containsKey(ConverterCollections.java:1502)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.containsKey(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.SafeNamedCache.containsKey(SafeNamedCache.CDB:1)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession.submissionExists(DefaultProcessingSession.java:616)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:98)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getResultAndDiscardSubmission(ActivityGridServiceImpl.java:131)
at com.foo.hi.support.gridprocessing.components.ActivityGridServiceImpl.getEmptySubmissions(ActivityGridServiceImpl.java:146)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityTaskLoaderControl.calculateHeadRoom(ActivityTaskLoaderControl.java:94)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityTaskLoaderControl.getBatchOfTaskIDs(ActivityTaskLoaderControl.java:68)
at com.foo.hi.activityprocessing.domain.internal.components.ActivityLoaderSession.loadNextBatch(ActivityLoaderSession.java:109)
at com.foo.hi.activityprocessing.domain.internal.ActivityLoaderTask.run(ActivityLoaderTask.java:83)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier{GridTaskProcessor}

,1,0x0AF194B700000154A5046D2D181A69EB}:Thread-601" daemon prio=10 tid=0x00007ff8e000c800 nid=0x4a96 waiting on condition [0x00007ff99f6f5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000007004a7d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier

{GridTaskProcessor},1,0x0AF194B700000154A5046D2D181A69EB}:Thread-600" daemon prio=10 tid=0x00007ff8e000b800 nid=0x4a81 waiting on condition [0x00007ff99f3f2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000007004a7d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DefaultTaskProcessor-TPSK:{Identifier{GridTaskProcessor}

,1,0x0AF194B700000154A5046D2D181A69EB}:Thread-599" daemon prio=10 tid=0x00007ff8e000b000 nid=0x4a80 runnable [0x00007ff99f8f6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at foo.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:759)
at foo.net.ns.Packet.receive(Packet.java:312)
at foo.net.ns.DataPacket.receive(DataPacket.java:106)
at foo.net.ns.NetInputStream.getNextPacket(NetInputStream.java:306)
at foo.net.ns.NetInputStream.read(NetInputStream.java:250)
at foo.net.ns.NetInputStream.read(NetInputStream.java:172)
at foo.net.ns.NetInputStream.read(NetInputStream.java:90)
at foo.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at foo.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at foo.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:452)
at foo.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:414)
at foo.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
at foo.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:603)
at foo.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:234)
at foo.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:55)
at foo.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:829)
at foo.jdbc.driver.fooStatement.executeMaybeDescribe(fooStatement.java:1049)
at foo.jdbc.driver.fooStatement.doExecuteWithTimeout(fooStatement.java:1270)
at foo.jdbc.driver.fooPreparedStatement.executeInternal(fooPreparedStatement.java:5010)
at foo.jdbc.driver.fooPreparedStatement.executeQuery(fooPreparedStatement.java:5070)

locked <0x00000006e497a660> (a foo.jdbc.driver.T4CConnection)
at foo.jdbc.driver.fooPreparedStatementWrapper.executeQuery(fooPreparedStatementWrapper.java:1677)
at wl.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:141)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:938)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:609)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1805)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:286)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:264)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:648)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2706)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2659)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:421)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1150)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1109)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:393)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1197)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2879)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1607)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1589)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1540)
at com.foo.hi.support.domain.service.PersistenceReader.search_aroundBody76(PersistenceReader.java:689)
at com.foo.hi.support.domain.service.PersistenceReader$AjcClosure77.run(PersistenceReader.java:1)
at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66)
at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70)
at com.foo.hi.support.domain.service.PersistenceReader.search(PersistenceReader.java:654)
at com.foo.hi.support.domain.service.PersistenceReader$$FastClassBySpringCGLIB$$7d6d00b1.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
at com.foo.hi.support.domain.service.PersistenceReader$$EnhancerBySpringCGLIB$$47a20ea6.search()
at com.foo.hi.support.domain.service.GenericRepositoryImpl.search(GenericRepositoryImpl.java:74)
at com.foo.hi.support.domain.search.SearchBuilder.doExecute(SearchBuilder.java:219)
at com.foo.hi.support.domain.search.SearchBuilder.execute(SearchBuilder.java:179)
at com.foo.hi.policies.domain.internal.repositories.MemberEnrollmentProductRepositoryImpl.fetchMemberEnrollmentProductsForPolicy(MemberEnrollmentProductRepositoryImpl.java:67)
at com.foo.hi.policies.domain.internal.entities.PolicyDomain.getMemberEnrollmentProductList(PolicyDomain.java:220)
at com.foo.hi.policies.domain.internal.calculatepremium.components.CalculatePremiumService.fetchCalculationCombinations(CalculatePremiumService.java:240)
at com.foo.hi.policies.domain.internal.calculatepremium.components.CalculatePremiumService.calculatePremium(CalculatePremiumService.java:102)
at com.foo.hi.policies.domain.internal.entities.PolicyDomain.calculatePremium(PolicyDomain.java:137)
at com.foo.hi.policies.domain.internal.calculatepremium.components.processors.CalculatePremiumActivityProcessor.execute(CalculatePremiumActivityProcessor.java:75)
at com.foo.hi.policies.domain.internal.activityprocessing.entities.PolicyActivityDomain.process(PolicyActivityDomain.java:38)
at com.foo.hi.activityprocessing.domain.internal.entities.BaseActivityDomain.execute(BaseActivityDomain.java:139)
at com.foo.hi.activityprocessing.service.internal.components.ActivityExecutorImpl.executeGroup(ActivityExecutorImpl.java:133)
at com.foo.hi.support.gridprocessing.task.GroupActivityTask.runTask(GroupActivityTask.java:32)
at com.foo.hi.support.gridprocessing.task.BaseActivityTask.run(BaseActivityTask.java:85)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.doExecuteTask(TaskRunner.java:316)
at com.foo.coherence.patterns.processing.taskprocessor.TaskRunner.run(TaskRunner.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"[STANDBY] ExecuteThread: '6' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa34006800 nid=0x408e in Object.wait() [0x00007ff8bfcfb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000717d002d0> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x0000000717d002d0> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"DynamicListenThread[policies_unicast_channel]" daemon prio=10 tid=0x00007ff95804e800 nid=0x408d runnable [0x00007ff8bfffe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)

locked <0x0000000717d004c0> (a java.lang.Object)
at wl.socket.wlServerSocket.accept(wlServerSocket.java:31)
at wl.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:532)
at wl.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:414)
at wl.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:745)

"DynamicListenThread[Default]" daemon prio=10 tid=0x00007ff958019000 nid=0x408c runnable [0x00007ff8bfefd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)

locked <0x0000000717d00668> (a java.lang.Object)
at wl.socket.wlServerSocket.accept(wlServerSocket.java:31)
at wl.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:532)
at wl.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:414)
at wl.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
at java.lang.Thread.run(Thread.java:745)

"MTUTimer" daemon prio=10 tid=0x00007ffa58338000 nid=0x4080 in Object.wait() [0x00007ffa446d2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000717d00760> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000717d00760> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"ReplicatedCache:foo.wls.internal.wsm:OWSMReplicatedCache" daemon prio=10 tid=0x00007ffa4cae6800 nid=0x407f waiting on condition [0x00007ff8bfdfc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000717d00818> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"wl.cluster.MessageReceiver" daemon prio=10 tid=0x00007ffa530fb800 nid=0x407e in Object.wait() [0x00007ff99c114000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006efb29c80> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:503)
at wl.cluster.messaging.internal.server.UnicastFragmentSocket.receive(UnicastFragmentSocket.java:107)
locked <0x00000006efb29c80> (a java.util.ArrayList)
at wl.cluster.FragmentSocketWrapper.receive(FragmentSocketWrapper.java:99)
at wl.cluster.ClusterMessagesManager.run(ClusterMessagesManager.java:468)
at java.lang.Thread.run(Thread.java:745)

"EventDispatcherThread-ADFApplication2" daemon prio=10 tid=0x00007ffa52d88800 nid=0x4077 waiting on condition [0x00007ff8bf8f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000718080070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at foo.mds.internal.notifications.EventDispatcherThread.run(EventDispatcherThread.java:75)

"JarIndex Timer" daemon prio=10 tid=0x00007ffa52c59000 nid=0x4075 in Object.wait() [0x00007ff8bf9f8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000717b81518> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000717b81518> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Dynamic Logic Re-load" daemon prio=10 tid=0x00007ffa52918000 nid=0x4071 waiting on condition [0x00007ff8bfbfa000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.foo.hi.common.domain.internal.dynlogic.components.DynamicLogicServiceImpl$ReloadRunnable.run(DynamicLogicServiceImpl.java:152)
at java.lang.Thread.run(Thread.java:745)

"AuditLoaderRunner" daemon prio=10 tid=0x00007ffa00003000 nid=0x406e in Object.wait() [0x00007ff8bf7f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000751f80000> (a foo.security.audit.service.AuditLoaderManager$Runner)
at foo.security.audit.service.AuditLoaderManager$Runner.sleep(AuditLoaderManager.java:332)
locked <0x0000000751f80000> (a foo.security.audit.service.AuditLoaderManager$Runner)
at foo.security.audit.service.AuditLoaderManager$Runner.run(AuditLoaderManager.java:364)

"Invocation:EntityCacheCoordination:EventDispatcher" daemon prio=10 tid=0x00007ffa518c8000 nid=0x3fcd in Object.wait() [0x00007ff99c417000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa80000> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa80000> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DefaultDispatchController.dispatcherThread" daemon prio=10 tid=0x00007ff8a0008800 nid=0x3f91 waiting on condition [0x00007ff99c619000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffa80128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:220)
at com.foo.coherence.patterns.processing.internal.DefaultDispatchController.run(DefaultDispatchController.java:300)
at java.lang.Thread.run(Thread.java:745)

"ReplicatedCache:ReplicatedServiceForProcessingPattern" daemon prio=10 tid=0x00007ff8a0005000 nid=0x3f8e waiting on condition [0x00007ff99e83c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffb80068> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedCacheForLiveObjects:EventDispatcher" daemon prio=10 tid=0x00007ff89c001000 nid=0x3f8a in Object.wait() [0x00007ff99c81b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa80280> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa80280> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedCacheForLiveObjects" daemon prio=10 tid=0x00007ffa52847000 nid=0x3f89 waiting on condition [0x00007ff8bfaf9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffa803a8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedCacheForSequenceGenerators:EventDispatcher" daemon prio=10 tid=0x00007ff8c4002000 nid=0x3f88 in Object.wait() [0x00007ff99c215000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffb80268> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffb80268> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedCacheForSequenceGenerators" daemon prio=10 tid=0x00007ffa52846000 nid=0x3f87 waiting on condition [0x00007ff99c91c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006f48656e8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternTaskMediators:EventDispatcher" daemon prio=10 tid=0x00007ff8c8002000 nid=0x3f86 in Object.wait() [0x00007ff99cb1e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffb80390> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffb80390> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternTaskMediators" daemon prio=10 tid=0x00007ffa52828000 nid=0x3f85 waiting on condition [0x00007ff99c518000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffa80670> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternSubmissions:EventDispatcher" daemon prio=10 tid=0x00007ff900002000 nid=0x3f84 in Object.wait() [0x00007ff99d326000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffb804b8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffb804b8> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternSubmissions" daemon prio=10 tid=0x00007ffa52827800 nid=0x3f83 waiting on condition [0x00007ff99d124000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffb805e0> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternSubmissionResults:EventDispatcher" daemon prio=10 tid=0x00007ff90c004800 nid=0x3f82 waiting for monitor entry [0x00007ff99d225000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.foo.coherence.patterns.processing.internal.DefaultProcessingSession$1.onMapEvent(DefaultProcessingSession.java:222)

waiting to lock <0x0000000717d00068> (a java.lang.Object)
at com.tangosol.util.MultiplexingMapListener.entryUpdated(MultiplexingMapListener.java:41)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:226)
at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:573)
at com.tangosol.coherence.component.util.SafeNamedCache.translateMapEvent(SafeNamedCache.CDB:7)
at com.tangosol.coherence.component.util.SafeNamedCache.entryUpdated(SafeNamedCache.CDB:1)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap$ProxyListener.dispatch(PartitionedCache.CDB:22)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap$ProxyListener.entryUpdated(PartitionedCache.CDB:1)
at com.tangosol.util.MapEvent.dispatch(MapEvent.java:270)
at com.tangosol.coherence.component.util.CacheEvent.run(CacheEvent.CDB:18)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onNotify(Service.CDB:28)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:DistributedServiceForProcessingPatternSubmissionResults" daemon prio=10 tid=0x00007ffa528e7800 nid=0x3f81 waiting on condition [0x00007ff99d427000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffa80938> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EventInvocationServiceWorker:4" daemon prio=10 tid=0x00007ff8f0009800 nid=0x3f80 in Object.wait() [0x00007ff99ca1d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EventInvocationServiceWorker:3" daemon prio=10 tid=0x00007ff8f0006000 nid=0x3f7f in Object.wait() [0x00007ff99d528000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EventInvocationServiceWorker:2" daemon prio=10 tid=0x00007ff8f0004800 nid=0x3f7e in Object.wait() [0x00007ff99d629000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EventInvocationServiceWorker:1" daemon prio=10 tid=0x00007ff8f0003000 nid=0x3f7d in Object.wait() [0x00007ff99ce21000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EventInvocationServiceWorker:0" daemon prio=10 tid=0x00007ff8f0002000 nid=0x3f7c in Object.wait() [0x00007ff99d92c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f48659b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"Invocation:EventInvocationService" daemon prio=10 tid=0x00007ffa528e6800 nid=0x3f7b waiting on condition [0x00007ff99cf22000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffb80ab8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EntityCacheCoordinationWorker:4" daemon prio=10 tid=0x00007ff93c006800 nid=0x3f7a in Object.wait() [0x00007ff99cd20000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EntityCacheCoordinationWorker:3" daemon prio=10 tid=0x00007ff93c005800 nid=0x3f79 in Object.wait() [0x00007ff99cc1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EntityCacheCoordinationWorker:2" daemon prio=10 tid=0x00007ff93c004800 nid=0x3f78 in Object.wait() [0x00007ff99dc2f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EntityCacheCoordinationWorker:1" daemon prio=10 tid=0x00007ff93c003800 nid=0x3f77 in Object.wait() [0x00007ff99d72a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"EntityCacheCoordinationWorker:0" daemon prio=10 tid=0x00007ff93c002800 nid=0x3f76 in Object.wait() [0x00007ff99e033000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865b10> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"Invocation:EntityCacheCoordination" daemon prio=10 tid=0x00007ffa52848000 nid=0x3f75 waiting on condition [0x00007ff99df32000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffb00238> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"OhiNonEntityCacheServiceWorker:3" daemon prio=10 tid=0x00007ff8fc009000 nid=0x3f74 in Object.wait() [0x00007ff99c71a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"OhiNonEntityCacheServiceWorker:2" daemon prio=10 tid=0x00007ff8fc007800 nid=0x3f73 in Object.wait() [0x00007ff99e336000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"OhiNonEntityCacheServiceWorker:1" daemon prio=10 tid=0x00007ff8fc006000 nid=0x3f72 in Object.wait() [0x00007ff99d82b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"OhiNonEntityCacheServiceWorker:0" daemon prio=10 tid=0x00007ff8fc005000 nid=0x3f71 in Object.wait() [0x00007ff99da2d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa00238> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:OhiNonEntityCacheService:EventDispatcher" daemon prio=10 tid=0x00007ff8fc004800 nid=0x3f70 in Object.wait() [0x00007ff99c316000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa00398> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa00398> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:OhiNonEntityCacheService" daemon prio=10 tid=0x00007ffa5288f800 nid=0x3f6f waiting on condition [0x00007ff99d023000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffb00420> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"SystemStatusCacheServiceWorker:1" daemon prio=10 tid=0x00007ff934005800 nid=0x3f6e in Object.wait() [0x00007ff99dd30000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865d58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865d58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"SystemStatusCacheServiceWorker:0" daemon prio=10 tid=0x00007ff934004800 nid=0x3f6d in Object.wait() [0x00007ff99e538000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006f4865d58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006f4865d58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$DaemonPool$Queue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onWait(DaemonPool.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid$DaemonPool$Daemon.onWait(Grid.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:SystemStatusCacheService:EventDispatcher" daemon prio=10 tid=0x00007ff934002000 nid=0x3f6c in Object.wait() [0x00007ff99db2e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffa004c0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffa004c0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:SystemStatusCacheService" daemon prio=10 tid=0x00007ffa5288e800 nid=0x3f6b waiting on condition [0x00007ff99de31000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006f4865eb8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:ConfigurationMigrationDistributedCache:EventDispatcher" daemon prio=10 tid=0x00007ff954002800 nid=0x3f6a in Object.wait() [0x00007ff99e235000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006ffb80f58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x00000006ffb80f58> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DistributedCache:ConfigurationMigrationDistributedCache" daemon prio=10 tid=0x00007ffa52843800 nid=0x3f69 waiting on condition [0x00007ff99e437000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000006ffa005e8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"ReplicatedCache:OhiPlanCacheService" daemon prio=10 tid=0x00007ffa526fd800 nid=0x3f67 waiting on condition [0x00007ff99e639000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x000000074e4321e8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"DmsThread-1" daemon prio=10 tid=0x00007ffa50e82000 nid=0x3f5f waiting on condition [0x00007ff99e133000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000007435fe3d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at foo.dms.util.DmsThreadFactory$1$1.run(DmsThreadFactory.java:55)
at java.security.AccessController.doPrivileged(Native Method)
at foo.dms.util.DmsThreadFactory$1.run(DmsThreadFactory.java:50)
at java.lang.Thread.run(Thread.java:745)

"[ACTIVE] ExecuteThread: '5' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa48001800 nid=0x3ef8 in Object.wait() [0x00007ff99e93d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b00000> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x0000000702b00000> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"LDAPConnThread-0 ldaps://slcao584.us.foo.com:8001" daemon prio=10 tid=0x00007ffa511f8000 nid=0x3ef6 runnable [0x00007ff99f1f0000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

locked <0x0000000702b02190> (a java.io.BufferedInputStream)
at netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
at netscape.ldap.LDAPConnThread.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)

"Timer-12" daemon prio=10 tid=0x00007ff9f000d000 nid=0x3ef3 in Object.wait() [0x00007ff99f4f3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b02370> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702b02370> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Timer-9" daemon prio=10 tid=0x00007ff9f0007800 nid=0x3ef0 in Object.wait() [0x00007ff99f7f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b9edd0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702b9edd0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Timer-7" daemon prio=10 tid=0x00007ff9f0004800 nid=0x3eee in Object.wait() [0x00007ff99f9f8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b024f0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702b024f0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Timer-6" daemon prio=10 tid=0x00007ff9f0003000 nid=0x3eed in Object.wait() [0x00007ff99faf9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b9ef50> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702b9ef50> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"wl.GCMonitor" daemon prio=10 tid=0x00007ffa6436e800 nid=0x3ee4 in Object.wait() [0x00007ff99fbfa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b025c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
locked <0x0000000702b025c0> (a java.lang.ref.ReferenceQueue$Lock)
at wl.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:86)
at wl.platform.GCMonitorThread.run(GCMonitorThread.java:62)

"NameService:TcpAcceptor:TcpProcessor" daemon prio=10 tid=0x00007ff994003000 nid=0x3ee3 runnable [0x00007ff99fcfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x0000000702b9f098> (a sun.nio.ch.Util$2)
locked <0x0000000702b9f0a8> (a java.util.Collections$UnmodifiableSet)
locked <0x0000000702b9f050> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector.select(MultiplexedSocketProvider.java:493)
locked <0x0000000702b9f0b8> (a java.util.HashSet)
locked <0x0000000702b9f0c8> (a com.foo.common.collections.UnmodifiableSetCollection)
locked <0x0000000702b9f0d8> (a com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.peer.acceptor.TcpAcceptor$TcpProcessor.onNotify(TcpAcceptor.CDB:12)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)

"NameService:TcpAcceptor" daemon prio=10 tid=0x00007ffa582e4000 nid=0x3ee2 in Object.wait() [0x00007ff99fdfc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b02658> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x0000000702b02658> (a com.tangosol.coherence.component.util.daemon.QueueProcessor$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"Invocation:Management" daemon prio=10 tid=0x00007ffa58213800 nid=0x3ee1 waiting on condition [0x00007ff99fefd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b9f228> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"Invocation:Management:EventDispatcher" daemon prio=10 tid=0x00007ffa5819e800 nid=0x3ee0 in Object.wait() [0x00007ffa452de000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b028b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x0000000702b028b0> (a com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service$EventDispatcher.onWait(Service.CDB:7)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"JMX Framework document pooling thread" daemon prio=10 tid=0x00007ff9ec0f9800 nid=0x3eb8 in Object.wait() [0x00007ff99fffe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b029d8> (a java.util.concurrent.ConcurrentLinkedQueue)
at foo.as.config.notification.filesystem.DocumentChangeNotifierImpl.waitForTaskOrNextCheckTime(DocumentChangeNotifierImpl.java:176)
locked <0x0000000702b029d8> (a java.util.concurrent.ConcurrentLinkedQueue)
at foo.as.config.notification.filesystem.DocumentChangeNotifierImpl.runOnce(DocumentChangeNotifierImpl.java:137)
at foo.as.config.notification.filesystem.DocumentChangeNotifierImpl.run(DocumentChangeNotifierImpl.java:129)
at java.lang.Thread.run(Thread.java:745)

"Cluster|Member(Id=1, Timestamp=2016-05-12 14:50:11.373, Address=10.241.148.183:27115, MachineId=6170, Location=site:,machine:slcao585,process:15667,member:policies_node11, Role=wlServer)" daemon prio=10 tid=0x00007ffa583bb000 nid=0x3eb7 runnable [0x00007ffa441cd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x0000000702b9f4e8> (a sun.nio.ch.Util$2)
locked <0x0000000702b9f4f8> (a java.util.Collections$UnmodifiableSet)
locked <0x0000000702b9f4a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector.select(MultiplexedSocketProvider.java:493)
locked <0x0000000702b9f508> (a java.util.HashSet)
locked <0x0000000702b9f518> (a com.foo.common.collections.UnmodifiableSetCollection)
locked <0x0000000702b9f528> (a com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector)
at com.tangosol.coherence.component.net.TcpRing.select(TcpRing.CDB:16)
locked <0x0000000702b9f528> (a com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService$TcpRing.await(ClusterService.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onWait(Grid.CDB:6)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"IpMonitor" daemon prio=10 tid=0x00007ffa583b9000 nid=0x3eb6 waiting on condition [0x00007ffa442ce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b02a98> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:58)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.IpMonitor.onWait(IpMonitor.CDB:4)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"PacketSpeaker" daemon prio=10 tid=0x00007ffa583ed800 nid=0x3eb5 in Object.wait() [0x00007ffa443cf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b9f840> (a com.tangosol.coherence.component.net.Cluster$PacketSpeaker$BundlingQueue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x0000000702b9f840> (a com.tangosol.coherence.component.net.Cluster$PacketSpeaker$BundlingQueue)
at com.tangosol.coherence.component.util.queue.ConcurrentQueue.await(ConcurrentQueue.CDB:12)
at com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:10)
at com.tangosol.coherence.component.util.Queue.remove(Queue.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketSpeaker.onNotify(PacketSpeaker.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)

"PacketPublisher" daemon prio=10 tid=0x00007ffa583eb800 nid=0x3eb4 waiting on condition [0x00007ffa444d0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b02bc0> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:54)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.onWait(PacketPublisher.CDB:2)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"PacketReceiver" daemon prio=10 tid=0x00007ffa583ea800 nid=0x3eb3 waiting on condition [0x00007ffa445d1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b02db8> (a com.foo.common.base.SingleWaiterMultiNotifier)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at com.foo.common.base.SingleWaiterMultiNotifier.await(SingleWaiterMultiNotifier.java:54)
at com.tangosol.coherence.component.util.queue.SingleConsumerQueue.await(SingleConsumerQueue.CDB:1)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketReceiver.onWait(PacketReceiver.CDB:2)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"PacketListener1P" daemon prio=10 tid=0x00007ffa583e9800 nid=0x3eb2 runnable [0x00007ffa448d4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)

locked <0x0000000702b4fe80> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
locked <0x0000000702b4fe80> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:816)
locked <0x0000000702b4feb0> (a java.net.DatagramPacket)
locked <0x0000000702b4fed8> (a java.net.DatagramSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:27)
locked <0x0000000702b4feb0> (a java.net.DatagramPacket)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:15)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)

"PacketListener1" daemon prio=10 tid=0x00007ffa583e9000 nid=0x3eb1 runnable [0x00007ffa449d5000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)

locked <0x0000000702bafa10> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
locked <0x0000000702bafa10> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:816)
locked <0x0000000702bafa40> (a java.net.DatagramPacket)
locked <0x0000000702bafa68> (a java.net.DatagramSocket)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:27)
locked <0x0000000702bafa40> (a java.net.DatagramPacket)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:15)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)

"SelectionService(channels=2, selector=MultiplexedSelector(sun.nio.ch.EPollSelectorImpl@4afb9871), id=1272269100)" daemon prio=10 tid=0x00007ffa583b3000 nid=0x3eb0 runnable [0x00007ffa44dd9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x0000000702b02fc8> (a sun.nio.ch.Util$2)
locked <0x0000000702b02fd8> (a java.util.Collections$UnmodifiableSet)
locked <0x0000000702b02f80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector.select(MultiplexedSocketProvider.java:493)
locked <0x0000000702b02fe8> (a java.util.HashSet)
locked <0x0000000702b02ff8> (a com.foo.common.collections.UnmodifiableSetCollection)
locked <0x0000000702b03008> (a com.foo.common.internal.net.MultiplexedSocketProvider$MultiplexedSelector)
at com.foo.common.internal.net.RunnableSelectionService.process(RunnableSelectionService.java:360)
at com.foo.common.internal.net.RunnableSelectionService.run(RunnableSelectionService.java:260)
at com.foo.common.internal.net.ResumableSelectionService.run(ResumableSelectionService.java:130)
at java.lang.Thread.run(Thread.java:745)

"foo.dfw.incident.FloodController" daemon prio=10 tid=0x00007ffa50af6000 nid=0x3ea8 in Object.wait() [0x00007ffa447d3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b500f8> (a java.lang.Object)
at foo.dfw.impl.incident.FloodController.run(FloodController.java:170)
locked <0x0000000702b500f8> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:745)

"foo.dfw.incident.IncidentCreatorThread" daemon prio=10 tid=0x00007ffa50af1800 nid=0x3ea5 in Object.wait() [0x00007ffa44ad6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702bafd20> (a java.util.ArrayList)
at java.lang.Object.wait(Object.java:503)
at foo.dfw.impl.incident.DiagnosticsDataExtractorImpl$IncidentCreatorThread.run(DiagnosticsDataExtractorImpl.java:2210)
locked <0x0000000702bafd20> (a java.util.ArrayList)
at java.lang.Thread.run(Thread.java:745)

"foo.dfw.impl.incident.ADRHelper - purge incidents" daemon prio=10 tid=0x00007ffa50af1000 nid=0x3ea3 in Object.wait() [0x00007ffa44cd8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b501a0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702b501a0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"foo.dfw.impl.incident - ADR Executor (created: Thu May 12 14:50:11 CEST 2016)" daemon prio=10 tid=0x00007ffa50ae9800 nid=0x3ea1 waiting on condition [0x00007ffa44bd7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b50260> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"DFW Diagnostic Dump Sampling - JavaClassHistogram" daemon prio=10 tid=0x00007ffa50ad2800 nid=0x3e9c in Object.wait() [0x00007ffa44eda000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702baff38> (a [B)
at foo.dfw.sampling.DumpSampling.run(DumpSampling.java:554)
locked <0x0000000702baff38> (a [B)

"DFW Diagnostic Dump Sampling - JVMThreadDump" daemon prio=10 tid=0x00007ffa50ad0800 nid=0x3e9b in Object.wait() [0x00007ffa44fdb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b50410> (a [B)
at foo.dfw.sampling.DumpSampling.run(DumpSampling.java:554)
locked <0x0000000702b50410> (a [B)

"ClickHistory Writer" daemon prio=10 tid=0x00007ffa50a95800 nid=0x3e99 in Object.wait() [0x00007ffa450dc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702baffd8> (a [B)
at foo.core.ojdl.logging.ClickHistoryWriter.waitForARecord(ClickHistoryWriter.java:178)
locked <0x0000000702baffd8> (a [B)
at foo.core.ojdl.logging.ClickHistoryWriter.run(ClickHistoryWriter.java:110)

"Logger@9238530 12.1.3.0.0" daemon prio=10 tid=0x00007ffa5836e000 nid=0x3e98 in Object.wait() [0x00007ffa453df000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702b50488> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Queue.await(Queue.CDB:7)
locked <0x0000000702b50488> (a com.tangosol.coherence.component.application.console.Coherence$Logger$Queue)
at com.tangosol.coherence.component.util.Daemon.onWait(Daemon.CDB:13)
at com.tangosol.coherence.component.util.daemon.QueueProcessor.onWait(QueueProcessor.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:40)
at java.lang.Thread.run(Thread.java:745)

"LogFlusher" daemon prio=10 tid=0x00007ffa50a83000 nid=0x3e96 waiting on condition [0x00007ffa451dd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at foo.core.ojdl.BufferedLogWriter$LogFlusher.run(BufferedLogWriter.java:401)
at java.lang.Thread.run(Thread.java:745)

"Thread-33" prio=10 tid=0x00007ffa642a2800 nid=0x3e90 waiting on condition [0x00007ffa454e0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000702b505f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at wl.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at wl.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:724)
at wl.store.internal.PersistentStoreImpl.synchronousFlush(PersistentStoreImpl.java:1124)
at wl.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:1116)
at java.lang.Thread.run(Thread.java:745)

"Timer-5" daemon prio=10 tid=0x00007ffa504b4000 nid=0x3e8f in Object.wait() [0x00007ffa45de3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702bb00e8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000702bb00e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"JPS FileChangeNotifier" daemon prio=10 tid=0x00007ffa4cbdf800 nid=0x3e8d waiting on condition [0x00007ffa45ee4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000705478440> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"Policy change scanner" daemon prio=10 tid=0x00007ffa4cbe1800 nid=0x3e8c waiting on condition [0x00007ffa45fe5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at foo.security.jps.az.internal.common.scanner.PolicyChangeScannerThread.run(PolicyChangeScannerThread.java:78)

"Runtime Authorization Event Notifier" daemon prio=10 tid=0x00007ffa4cbd8000 nid=0x3e8b in Object.wait() [0x00007ffa460e6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702bb0280> (a foo.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
at java.lang.Object.wait(Object.java:503)
at foo.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.run(RuntimeEventNotifierImpl.java:232)
locked <0x0000000702bb0280> (a foo.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
at java.lang.Thread.run(Thread.java:745)

"Thread-28" daemon prio=10 tid=0x00007ffa4caae000 nid=0x3e84 waiting on condition [0x00007ffa462e8000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at foo.security.audit.config.dynamic.persistence.internal.ldap.ScanningRunner.run(ScanningRunner.java:66)

"Timer-3" daemon prio=10 tid=0x00007ffa4d184800 nid=0x3e82 in Object.wait() [0x00007ffa463e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000702bb0320> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
locked <0x0000000702bb0320> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"foo.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" daemon prio=10 tid=0x00007ffa4cfec000 nid=0x3e81 in Object.wait() [0x00007ffa464ea000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470d08> (a foo.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)
at foo.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:340)
locked <0x0000000705470d08> (a foo.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)

"Abandoned connection cleanup thread" daemon prio=10 tid=0x00007ffa4cdbe800 nid=0x3e80 in Object.wait() [0x00007ffa465eb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470d70> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
locked <0x0000000705470d70> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at com.mysql.jdbc.NonRegisteringDriver$1.run(NonRegisteringDriver.java:93)

"Thread-26" daemon prio=10 tid=0x00007ffa642c7000 nid=0x3e1e runnable [0x00007ffa466ec000]
java.lang.Thread.State: RUNNABLE
at wl.nodemanager.NMService.rotateLogFiles0(Native Method)
at wl.nodemanager.NMService.access$1300(NMService.java:61)
at wl.nodemanager.NMService$1.run(NMService.java:507)

"NM Reaper" daemon prio=10 tid=0x00007ffa642c3800 nid=0x3e1d in Object.wait() [0x00007ffa467ed000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470dd8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x0000000705470dd8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Thread-24" prio=10 tid=0x00007ffa642b7000 nid=0x3e1c waiting on condition [0x00007ffa46bf4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00000007054785c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at wl.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
at wl.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:724)
at wl.store.internal.PersistentStoreImpl.synchronousFlush(PersistentStoreImpl.java:1124)
at wl.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:1116)
at java.lang.Thread.run(Thread.java:745)

"DoSManager" daemon prio=10 tid=0x00007ffa20003800 nid=0x3e13 waiting on condition [0x00007ffa46af3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.octetstring.vde.DoSManager.run(DoSManager.java:433)

"VDE Transaction Processor Thread" daemon prio=10 tid=0x00007ffa4c0dc800 nid=0x3e11 in Object.wait() [0x00007ffa46cf5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470c80> (a com.octetstring.vde.backend.standard.TransactionProcessor)
at java.lang.Object.wait(Object.java:503)
at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
locked <0x0000000705470c80> (a com.octetstring.vde.backend.standard.TransactionProcessor)
at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"[STANDBY] ExecuteThread: '4' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa38002000 nid=0x3e10 in Object.wait() [0x00007ffa472fa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x000000070547b418> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x000000070547b418> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"ExecuteThread: '3' for queue: 'wl.socket.Muxer'" daemon prio=10 tid=0x00007ffa4c049000 nid=0x3e0f runnable [0x00007ffa473fb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x0000000705478800> (a sun.nio.ch.Util$2)
locked <0x0000000705478810> (a java.util.Collections$UnmodifiableSet)
locked <0x00000007054787b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at wl.socket.NIOSocketMuxer.selectFrom(NIOSocketMuxer.java:529)
at wl.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:458)
at wl.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
at wl.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at wl.kernel.ExecuteThread.execute(ExecuteThread.java:147)
at wl.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '2' for queue: 'wl.socket.Muxer'" daemon prio=10 tid=0x00007ffa4c047800 nid=0x3e0e runnable [0x00007ffa474fc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x000000070546efb8> (a sun.nio.ch.Util$2)
locked <0x000000070546efc8> (a java.util.Collections$UnmodifiableSet)
locked <0x000000070546ef70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at wl.socket.NIOSocketMuxer.selectFrom(NIOSocketMuxer.java:529)
at wl.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:458)
at wl.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
at wl.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at wl.kernel.ExecuteThread.execute(ExecuteThread.java:147)
at wl.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '1' for queue: 'wl.socket.Muxer'" daemon prio=10 tid=0x00007ffa4c040000 nid=0x3e0d runnable [0x00007ffa475fd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x0000000705472c58> (a sun.nio.ch.Util$2)
locked <0x0000000705472c68> (a java.util.Collections$UnmodifiableSet)
locked <0x0000000705472c10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at wl.socket.NIOSocketMuxer.selectFrom(NIOSocketMuxer.java:529)
at wl.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:458)
at wl.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
at wl.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at wl.kernel.ExecuteThread.execute(ExecuteThread.java:147)
at wl.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '0' for queue: 'wl.socket.Muxer'" daemon prio=10 tid=0x00007ffa4c043000 nid=0x3e0c runnable [0x00007ffaa091a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

locked <0x00000007054789b8> (a sun.nio.ch.Util$2)
locked <0x00000007054789c8> (a java.util.Collections$UnmodifiableSet)
locked <0x0000000705478970> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at wl.socket.NIOSocketMuxer.selectFrom(NIOSocketMuxer.java:529)
at wl.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:458)
at wl.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
at wl.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
at wl.kernel.ExecuteThread.execute(ExecuteThread.java:147)
at wl.kernel.ExecuteThread.run(ExecuteThread.java:119)

"[STANDBY] ExecuteThread: '3' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa4c004800 nid=0x3e0b in Object.wait() [0x00007ffaa0a1b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e97009c0> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x00000006e97009c0> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"[STANDBY] ExecuteThread: '2' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa4c002000 nid=0x3e0a in Object.wait() [0x00007ffaa0819000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705478a48> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x0000000705478a48> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"[STANDBY] ExecuteThread: '1' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa4c001000 nid=0x3e09 in Object.wait() [0x00007ffaa0b1c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x000000070546df98> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x000000070546df98> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"wl.timers.TimerThread" daemon prio=10 tid=0x00007ffa503d1000 nid=0x3e07 in Object.wait() [0x00007ffa94168000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470dc8> (a java.lang.Object)
at wl.timers.internal.TimerThread.doWait(TimerThread.java:114)
locked <0x0000000705470dc8> (a java.lang.Object)
at wl.timers.internal.TimerThread.access$000(TimerThread.java:22)
at wl.timers.internal.TimerThread$Thread.run(TimerThread.java:290)

"wl.time.TimeEventGenerator" daemon prio=10 tid=0x00007ffa503c5000 nid=0x3e06 in Object.wait() [0x00007ffa9436a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x0000000705470d80> (a wl.time.common.internal.TimeTable)
at wl.time.common.internal.TimeTable.snooze(TimeTable.java:286)
locked <0x0000000705470d80> (a wl.time.common.internal.TimeTable)
at wl.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)
at java.lang.Thread.run(Thread.java:745)

"[ACTIVE] ExecuteThread: '0' for queue: 'wl.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ffa50065800 nid=0x3e04 in Object.wait() [0x00007ffaa01da000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e4339900> (a wl.work.ExecuteThread)
at java.lang.Object.wait(Object.java:503)
at wl.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
locked <0x00000006e4339900> (a wl.work.ExecuteThread)
at wl.work.ExecuteThread.run(ExecuteThread.java:281)

"Timer-2" daemon prio=10 tid=0x00007ffa5005e000 nid=0x3e03 in Object.wait() [0x00007ffaa02db000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e95be1d0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
locked <0x00000006e95be1d0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Timer-1" daemon prio=10 tid=0x00007ffa58019000 nid=0x3e01 in Object.wait() [0x00007ffaa0718000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e78111d8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
locked <0x00000006e78111d8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Timer-0" daemon prio=10 tid=0x00007ffc017f5000 nid=0x3dec in Object.wait() [0x00007ffaa0c1d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e7811d80> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
locked <0x00000006e7811d80> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007ffc016f5000 nid=0x3de7 runnable [0x00007ffaa0e1f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
at java.lang.Thread.run(Thread.java:745)

"RMI TCP Accept-27215" daemon prio=10 tid=0x00007ffc016ef000 nid=0x3de6 runnable [0x00007ffaa0f20000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
at java.lang.Thread.run(Thread.java:745)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007ffc016df000 nid=0x3de4 runnable [0x00007ffaa1021000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" daemon prio=10 tid=0x00007ffc00caf800 nid=0x3dde runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007ffc00cad000 nid=0x3ddd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007ffc00c9a800 nid=0x3ddc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"VM JFR Buffer Thread" daemon prio=10 tid=0x00007ffc00c91800 nid=0x3ddb waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"JFR request timer" daemon prio=10 tid=0x00007ffc00bbc800 nid=0x3dd9 in Object.wait() [0x00007ffaa322c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e7812830> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
locked <0x00000006e7812830> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"Signal Dispatcher" daemon prio=10 tid=0x00007ffc00bb6000 nid=0x3dd6 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007ffc00bb4000 nid=0x3dd5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007ffc00b7d800 nid=0x3dd3 in Object.wait() [0x00007ffad0198000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e7812810> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
locked <0x00000006e7812810> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007ffc00b7b800 nid=0x3dd1 in Object.wait() [0x00007ffad0299000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e7812800> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
locked <0x00000006e7812800> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007ffc0000d000 nid=0x3d36 in Object.wait() [0x00007ffc0705d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

waiting on <0x00000006e7813280> (a wl.t3.srvr.T3Srvr)
at java.lang.Object.wait(Object.java:503)
at wl.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1130)
locked <0x00000006e7813280> (a wl.t3.srvr.T3Srvr)
at wl.t3.srvr.T3Srvr.run(T3Srvr.java:538)
at wl.Server.main(Server.java:65)

"VM Thread" prio=10 tid=0x00007ffc00b77000 nid=0x3dcf runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007ffc00022800 nid=0x3d38 runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007ffc00024800 nid=0x3d3a runnable

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007ffc00026800 nid=0x3d3c runnable

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007ffc00028800 nid=0x3d3e runnable

"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007ffc0002a800 nid=0x3d40 runnable

"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007ffc0002c800 nid=0x3d42 runnable

"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007ffc0002e000 nid=0x3d44 runnable

"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007ffc00030000 nid=0x3d46 runnable

"Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007ffc00032000 nid=0x3d48 runnable

"Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007ffc00034000 nid=0x3d4a runnable

"Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007ffc00036000 nid=0x3d4c runnable

"Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007ffc00038000 nid=0x3d4e runnable

"Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007ffc00039800 nid=0x3d50 runnable

"Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007ffc0003b800 nid=0x3d52 runnable

"Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007ffc0003d800 nid=0x3d54 runnable

"Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007ffc0003f800 nid=0x3d56 runnable

"Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007ffc00041800 nid=0x3d57 runnable

"Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007ffc00043800 nid=0x3d59 runnable

"Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007ffc00045000 nid=0x3d5b runnable

"Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007ffc00047000 nid=0x3d5d runnable

"Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007ffc00049000 nid=0x3d5e runnable

"Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007ffc0004b000 nid=0x3d60 runnable

"Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007ffc0004d000 nid=0x3d62 runnable

"Gang worker#23 (Parallel GC Threads)" prio=10 tid=0x00007ffc0004f000 nid=0x3d64 runnable

"Gang worker#24 (Parallel GC Threads)" prio=10 tid=0x00007ffc00051000 nid=0x3d66 runnable

"Gang worker#25 (Parallel GC Threads)" prio=10 tid=0x00007ffc00052800 nid=0x3d68 runnable

"Gang worker#26 (Parallel GC Threads)" prio=10 tid=0x00007ffc00054800 nid=0x3d6a runnable

"Gang worker#27 (Parallel GC Threads)" prio=10 tid=0x00007ffc00056800 nid=0x3d6c runnable

"Gang worker#28 (Parallel GC Threads)" prio=10 tid=0x00007ffc00058800 nid=0x3d6e runnable

"Gang worker#29 (Parallel GC Threads)" prio=10 tid=0x00007ffc0005a800 nid=0x3d70 runnable

"Gang worker#30 (Parallel GC Threads)" prio=10 tid=0x00007ffc0005c800 nid=0x3d72 runnable

"Gang worker#31 (Parallel GC Threads)" prio=10 tid=0x00007ffc0005e000 nid=0x3d74 runnable

"Gang worker#32 (Parallel GC Threads)" prio=10 tid=0x00007ffc00060000 nid=0x3d77 runnable

"G1 Main Concurrent Mark GC Thread" prio=10 tid=0x00007ffc000b4800 nid=0x3dbd runnable

"Gang worker#0 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001c3800 nid=0x3dc1 runnable

"Gang worker#1 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001c5800 nid=0x3dc3 runnable

"Gang worker#2 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001c7800 nid=0x3dc5 runnable

"Gang worker#3 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001c9800 nid=0x3dc7 runnable

"Gang worker#4 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001cb800 nid=0x3dc9 runnable

"Gang worker#5 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001cd800 nid=0x3dca runnable

"Gang worker#6 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001cf000 nid=0x3dcd runnable

"Gang worker#7 (G1 Parallel Marking Threads)" prio=10 tid=0x00007ffc001d1000 nid=0x3dce runnable

"G1 Concurrent Refinement Thread#0" prio=10 tid=0x00007ffc000a6000 nid=0x3dbb runnable

"G1 Concurrent Refinement Thread#1" prio=10 tid=0x00007ffc000a4000 nid=0x3db9 runnable

"G1 Concurrent Refinement Thread#2" prio=10 tid=0x00007ffc000a2000 nid=0x3db7 runnable

"G1 Concurrent Refinement Thread#3" prio=10 tid=0x00007ffc000a0000 nid=0x3db4 runnable

"G1 Concurrent Refinement Thread#4" prio=10 tid=0x00007ffc0009e000 nid=0x3db2 runnable

"G1 Concurrent Refinement Thread#5" prio=10 tid=0x00007ffc0009c800 nid=0x3db0 runnable

"G1 Concurrent Refinement Thread#6" prio=10 tid=0x00007ffc0009a800 nid=0x3dae runnable

"G1 Concurrent Refinement Thread#7" prio=10 tid=0x00007ffc00098800 nid=0x3dac runnable

"G1 Concurrent Refinement Thread#8" prio=10 tid=0x00007ffc00096800 nid=0x3daa runnable

"G1 Concurrent Refinement Thread#9" prio=10 tid=0x00007ffc00094800 nid=0x3da9 runnable

"G1 Concurrent Refinement Thread#10" prio=10 tid=0x00007ffc00092800 nid=0x3da7 runnable

"G1 Concurrent Refinement Thread#11" prio=10 tid=0x00007ffc00090800 nid=0x3da5 runnable

"G1 Concurrent Refinement Thread#12" prio=10 tid=0x00007ffc0008e800 nid=0x3da3 runnable

"G1 Concurrent Refinement Thread#13" prio=10 tid=0x00007ffc0008c800 nid=0x3da1 runnable

"G1 Concurrent Refinement Thread#14" prio=10 tid=0x00007ffc0008a800 nid=0x3d9f runnable

"G1 Concurrent Refinement Thread#15" prio=10 tid=0x00007ffc00089000 nid=0x3d9d runnable

"G1 Concurrent Refinement Thread#16" prio=10 tid=0x00007ffc00087000 nid=0x3d9b runnable

"G1 Concurrent Refinement Thread#17" prio=10 tid=0x00007ffc00085000 nid=0x3d99 runnable

"G1 Concurrent Refinement Thread#18" prio=10 tid=0x00007ffc00083000 nid=0x3d97 runnable

"G1 Concurrent Refinement Thread#19" prio=10 tid=0x00007ffc00081000 nid=0x3d95 runnable

"G1 Concurrent Refinement Thread#20" prio=10 tid=0x00007ffc0007f000 nid=0x3d94 runnable

"G1 Concurrent Refinement Thread#21" prio=10 tid=0x00007ffc0007d000 nid=0x3d92 runnable

"G1 Concurrent Refinement Thread#22" prio=10 tid=0x00007ffc0007b000 nid=0x3d90 runnable

"G1 Concurrent Refinement Thread#23" prio=10 tid=0x00007ffc00079000 nid=0x3d8d runnable

"G1 Concurrent Refinement Thread#24" prio=10 tid=0x00007ffc00077000 nid=0x3d8c runnable

"G1 Concurrent Refinement Thread#25" prio=10 tid=0x00007ffc00075800 nid=0x3d8a runnable

"G1 Concurrent Refinement Thread#26" prio=10 tid=0x00007ffc00073800 nid=0x3d88 runnable

"G1 Concurrent Refinement Thread#27" prio=10 tid=0x00007ffc00071800 nid=0x3d86 runnable

"G1 Concurrent Refinement Thread#28" prio=10 tid=0x00007ffc0006f800 nid=0x3d84 runnable

"G1 Concurrent Refinement Thread#29" prio=10 tid=0x00007ffc0006d800 nid=0x3d82 runnable

"G1 Concurrent Refinement Thread#30" prio=10 tid=0x00007ffc0006b800 nid=0x3d80 runnable

"G1 Concurrent Refinement Thread#31" prio=10 tid=0x00007ffc00069800 nid=0x3d7e runnable

"G1 Concurrent Refinement Thread#32" prio=10 tid=0x00007ffc00067800 nid=0x3d7b runnable

"G1 Concurrent Refinement Thread#33" prio=10 tid=0x00007ffc00065800 nid=0x3d7a runnable

"VM Periodic Task Thread" prio=10 tid=0x00007ffc016f7000 nid=0x3de9 waiting on condition

JNI global references: 552

brianoliver · 2016-06-30T17:30:45Z

This issue was imported from JIRA COHINC-145

brianoliver added Type: Improvement Priority: Major Module: Processing Pattern labels Jun 30, 2016

brianoliver self-assigned this Jun 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Processing Pattern should use non-Coherence threads #145

Processing Pattern should use non-Coherence threads #145

brianoliver commented Aug 27, 2015

brianoliver commented Aug 27, 2015

brianoliver commented May 23, 2016

brianoliver commented Jun 30, 2016

Processing Pattern should use non-Coherence threads #145

Processing Pattern should use non-Coherence threads #145

Comments

brianoliver commented Aug 27, 2015

brianoliver commented Aug 27, 2015

brianoliver commented May 23, 2016

brianoliver commented Jun 30, 2016