Logstash plugin crashes at high volume

[andrew-morris]

Reading the error message this seems that it may have been an API timeout issue.

@Obsecurus @guillermo-menjivar

warning: thread "[reindex-sumologic]>worker14" terminated with exception (report_on_exception is true):
[ERROR] 2020-06-16 23:23:55.675 [[reindex-sumologic]>worker9] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
textPayload: "org.jruby.exceptions.RuntimeError: (OpenTimeout) execution expired
    at org.jruby.ext.socket.RubyTCPSocket.initialize(org/jruby/ext/socket/RubyTCPSocket.java:144) ~[jruby-complete-9.2.8.0.jar:?]
    at org.jruby.RubyIO.open(org/jruby/RubyIO.java:1156) ~[jruby-complete-9.2.8.0.jar:?]
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.net.http.connect(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:941) ~[?:?]
    at org.jruby.ext.timeout.Timeout.timeout(org/jruby/ext/timeout/Timeout.java:99) ~[jruby-complete-9.2.8.0.jar:?]
    at org.jruby.ext.timeout.Timeout.timeout(org/jruby/ext/timeout/Timeout.java:75) ~[jruby-complete-9.2.8.0.jar:?]
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.net.http.connect(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:939) ~[?:?]
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.net.http.do_start(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924) ~[?:?]
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.net.http.start(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913) ~[?:?]
    at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.net.http.start(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:609) ~[?:?]
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_greynoise_minus_0_dot_1_dot_7.lib.logstash.filters.greynoise.get_enterprise(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-greynoise-0.1.7/lib/logstash/filters/greynoise.rb:89) ~[?:?]
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_greynoise_minus_0_dot_1_dot_7.lib.logstash.filters.greynoise.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-greynoise-0.1.7/lib/logstash/filters/greynoise.rb:121) ~[?:?]
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?]
    at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1800) ~[jruby-complete-9.2.8.0.jar:?]
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
    at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?]
    at RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:250) ~[?:?]
"
"[FATAL] 2020-06-16 23:37:56.000 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.jruby.exceptions.RuntimeError: (OpenTimeout) execution expired, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:250)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}

[nicksherron]

Hey @andrew-morris could you help describe how I might reproduce this and would you happen to know the gem version?

[mlmurray]

Hey guys - can confirm that we're seeing this exact same error in our kubernetes instance.

Is it possible that because Net::HTTP isn't threadsafe, it's dying when we're running multiple concurrent requests?