10x speed differences between v5.2.2 and v5.2.3

[xia0pin9]

Logstash information:

Please include the following information:

1. Logstash version: 8.2.0
2. Logstash installation source: docker
3. How is Logstash being run: docker
4. How was the Logstash Plugin installed: ./bin/logstash-plugin install --version xxx logstash-output-http

JVM (e.g. java -version): using the bundled JDK

OS version (uname -a if on a Unix-like system): Linux hostname 4.15.0-1065-aws #69-Ubuntu SMP Thu Mar 26 02:17:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
We have a logstash pipeline that reads from kafka and write to https endpoint. After upgrading logstash version, I noticed that the pipeline throughput was dramatically reduced. The logstash node stats api showed that majority time were spent in http output plugin. After manually uninstall logstash-filter-http, logstash-input-http_poller, logstash-output-http, and gradually upgrade logstash-output-http version, I noticed the performance degradation started from v5.2.3.

Stats for v5.2.2:

      "events" : {
        "out" : 328875,
        "filtered" : 328875,
        "duration_in_millis" : 1474697,
        "queue_push_duration_in_millis" : 115194,
        "in" : 331876
      }

        "outputs" : [ {
          "events" : {
            "out" : 328875,
            "duration_in_millis" : 1474551,
            "in" : 330375
          },
          "name" : "http"
        } ]

Stats for v5.2.3:

      "events" : {
        "out" : 21000,
        "queue_push_duration_in_millis" : 119697,
        "filtered" : 21000,
        "in" : 24001,
        "duration_in_millis" : 1437976
      }

        "outputs" : [ {
          "name" : "http",
          "events" : {
            "out" : 21000,
            "in" : 22500,
            "duration_in_millis" : 1437954
          }
        } ]

Steps to reproduce:

pipeline config:

input {
  kafka {
    codec            => "json"
    consumer_threads => 2
 }
}

filter {
}

output {
  http {
    url => "https_target"
    http_method => post
    format => "json"
    content_type => "application/json"
    #ssl_verification_mode => "none"
    pool_max => 200
    pool_max_per_route => 200
  }
}

logstash.yml:

cat logstash.yml
log.level: info
pipeline.batch.delay: 5
pipeline.workers: 12