job logs stopped being uploaded to S3

[makered]

I have a two node Rundeck cluster and using the S3 plugin. Recently I noticed that as of about 10 days ago logs stopped being delivered to S3 and one of the Rundeck nodes cannot access logs in the UI (logs are possibly buffered of the local filesystem of the other node).

Our users started scheduling more jobs lately, some running fairly frequently (every minute) and I am wondering if this might cause a inflow of execution logs that the S3 plugin cannot keep up with.

Once I restarted one of the nodes, it spends a long time (over 3 hrs) initially re-queueing logs with messages similar to:

2016-03-19 01:15:47,794 [main] INFO grails.app.services.rundeck.services.LogFileStorageService - re-queueing incomplete log storage request for execution 546710

I checked Rundeck logs and there do not seem to be any clues there as to why this is occurring. Are there any recommended steps I should follow to troubleshoot the plugin?

Thank you

[lorenzoaiello]

Any updates on this? We are running into the same issue.

[gschueler]

what rundeck version are you using?

[lorenzoaiello]

Rundeck 2.6.9-1

[tkant]

Hi @gschueler, wondering if we have any updates on this issue? A user of RUNDECK ENTERPRISE 3.1.2-20190927 version.