Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline crashes with undefined method `each' for nil:NilClass error in event filter method #191

Open
erhudy opened this issue Jun 28, 2024 · 0 comments
Labels

Comments

@erhudy
Copy link

erhudy commented Jun 28, 2024

Logstash information:

  1. Logstash version: 7.17.3
  2. Installation source: Elastic container image
  3. Being run: via Docker
  4. Included in container image (Grok 4.4.1)
  5. JVM version: whatever is in the container image (11?)
  6. OS: RHEL 7.7

Description of the problem including expected versus actual behavior: Our Logstash pipelines sometimes crash with a particular error in the Grok plugin:

{
    "level": "ERROR",
    "loggerName": "logstash.javapipeline",
    "timeMillis": 1719522558635,
    "thread": "[beats-plain]>worker11",
    "logEvent": {
        "message": "Pipeline worker error, the pipeline will be stopped",
        "pipeline_id": "beats-plain",
        "error": "(NoMethodError) undefined method `each' for nil:NilClass",
        "exception": {
            "metaClass": {
                "metaClass": {
                    "exception": "Java::OrgJrubyExceptions::NoMethodError",
                    "backtrace": [
                        "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_grok_minus_4_dot_4_dot_1.lib.logstash.filters.grok.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.1/lib/logstash/filters/grok.rb:300)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)",
                        "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)",
                        "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299)"
                    ],
                    "thread": "#<Thread:0x4cc678c3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:53 sleep>"
                }
            }
        }
    }
}

I suspect that there is a race condition of some sort based on the fact that we can have Logstash instances go weeks or even longer without suffering from this problem and yet relatively low-traffic instances will exhibit the pipeline crash and require a Logstash restart (not that this seems directly tied to low traffic, it happens on both low- and high-traffic instances). When it does happen, we will get a couple of these tracebacks on different threads in Logstash and then the pipeline grinds to a halt and we get an alert that it is no longer passing events and we have to come and restart Logstash.

Steps to reproduce: Unfortunately there is no known reproducer for the bug, it just happens randomly.

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it:

If I had one I would put it here.

@erhudy erhudy added the bug label Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant