Cannot create many newspaper processes

[matthias-ronge]

I tried to create a newspaper with 14.000 issues. The first around 1000 issues have been created without errors, then the time to create an issues (in seconds) rises…

…and finally the thread fails with the following exception:

[ERROR] 2021-02-26 18:48:15.698 [Erzeuge Zeitungsvorgänge: BudiFrunA_020166176] EmptyTask - Request cannot be executed; I/O reactor status: STOPPED
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46) ~[httpcore-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343) ~[elasticsearch-rest-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:325) ~[elasticsearch-rest-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[elasticsearch-rest-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191) ~[elasticsearch-rest-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:396) ~[elasticsearch-rest-high-level-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:382) ~[elasticsearch-rest-high-level-client-5.6.16.jar:5.6.16]
        at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:267) ~[elasticsearch-rest-high-level-client-5.6.16.jar:5.6.16]
        at org.kitodo.data.elasticsearch.index.IndexRestClient.addDocument(IndexRestClient.java:91) ~[kitodo-data-management-3.2.1-SNAPSHOT.jar:?]
        at org.kitodo.data.elasticsearch.index.Indexer.performSingleRequest(Indexer.java:72) ~[kitodo-data-management-3.2.1-SNAPSHOT.jar:?]
        at org.kitodo.production.services.data.base.SearchService.saveToIndex(SearchService.java:195) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.ProcessService.saveToIndex(ProcessService.java:265) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.ProcessService.saveToIndex(ProcessService.java:161) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.base.SearchService.save(SearchService.java:281) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.ProcessService.save(ProcessService.java:254) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.ProcessService.save(ProcessService.java:256) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.services.data.ProcessService.save(ProcessService.java:256) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.process.NewspaperProcessesGenerator.createProcess(NewspaperProcessesGenerator.java:522) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.process.NewspaperProcessesGenerator.nextStep(NewspaperProcessesGenerator.java:296) ~[classes/:3.2.1-SNAPSHOT]
        at org.kitodo.production.helper.tasks.GeneratesNewspaperProcessesThread.run(GeneratesNewspaperProcessesThread.java:79) ~[classes/:3.2.1-SNAPSHOT]

[matthias-ronge]

Adding additional logging revealed that this seems not to be an ElasticSearch problem (unlike the stacktrace above may indicate), indexing times do not increase.

Instead, I got an out of memory error this time:

16-Mar-2021 19:56:10.677 SCHWERWIEGEND [Catalina-utility-2] org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async timeouts
        java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
                at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
                at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
                at org.apache.coyote.AbstractProtocol.startAsyncTimeout(AbstractProtocol.java:615)
                at org.apache.coyote.AbstractProtocol$1.run(AbstractProtocol.java:598)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
                at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.base/java.lang.Thread.run(Thread.java:834)
        Caused by: java.lang.OutOfMemoryError: Java heap space

[andre-hohmann]

@matthias-ronge : Could this be reason for the following issue?

• Performance problem: Creating newspaper processes - creation of processes is very slow #4760

[matthias-ronge]

Yes, I think this is the same problem.