Logstash on search node stops processing Redis queues an hour or so after each reboot

[Ximalas]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

384 GB

Storage for /

314 GB

Storage for /nsm

7672 GB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

From /opt/so/log/logstash/logstash.log on the search node:

:error=>{"type"=>"illegal_state_exception", "reason"=>"Pipeline processor configured for non-existent pipeline [suricata.]"}}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

A couple of questions. Are you using Suricata as metadata and what is the rest of that error log?

[Ximalas]

Yes, I'm using Suricata for metadata.

Here are the last two lines of the current log file on the search node:

[2025-01-22T20:22:27,127][INFO ][logstash.outputs.elasticsearch] Retrying failed action {:status=>500, :action=>["create", {:_id=>nil, :_index=>"logs-suricata-so", :routing=>nil, :pipeline=>"suricata.common"}, {"data_stream"=>{"namespace"=>"so", "dataset"=>"suricata", "type"=>"logs"}, "ecs"=>{"version"=>"8.0.0"}, "metadata"=>{"input_id"=>"logfile-logs-9c1901e7-ea35-49ec-9fad-f73b4d71886e", "beat"=>"filebeat", "raw_index"=>"logs-suricata-so", "pipeline"=>"suricata.common", "input"=>{"beats"=>{"host"=>{"ip"=>"[forward node's public IPv4 address]"}}}, "version"=>"8.14.3", "type"=>"_doc", "stream_id"=>"logfile-log.logs-9c1901e7-ea35-49ec-9fad-f73b4d71886e"}, "@version"=>"1", "host"=>{"architecture"=>"x86_64", "ip"=>["[forward node's public IPv4 address]", "[autogenerated public IPv6 address]", "fe80::42a6:b7ff:fec1:f778", "172.17.1.1", "172.17.0.1"], "mac"=>["00-0A-F7-11-7C-3C", "00-0A-F7-11-7C-3D", "00-0A-F7-11-7C-3E", "00-0A-F7-11-7C-3F", "00-0A-F7-11-7C-5C", "00-0A-F7-11-7C-5D", "00-0A-F7-11-7C-5E", "00-0A-F7-11-7C-5F", "02-42-1F-CF-2D-E6", "02-42-6D-B4-CA-D3", "2E-09-35-C4-16-84", "40-A6-B7-36-41-A8", "40-A6-B7-36-41-AA", "40-A6-B7-36-41-AB", "40-A6-B7-C1-F7-78", "40-A6-B7-C1-F7-79", "4A-8F-8A-1B-EC-10", "5A-85-3B-FE-A1-31", "8E-08-2F-80-9F-54", "90-B1-1C-4B-14-97", "90-B1-1C-4B-14-98", "90-B1-1C-4B-14-99", "90-B1-1C-4B-14-9A", "AE-ED-B0-DC-62-34", "F2-D4-6B-75-33-9D"], "name"=>"so-fn01", "id"=>"b23a6d7387d844138deb33843bdd39ae", "containerized"=>false, "os"=>{"family"=>"redhat", "name"=>"Oracle Linux Server", "kernel"=>"5.15.0-304.171.4.1.el9uek.x86_64", "version"=>"9.5", "type"=>"linux", "platform"=>"ol"}, "hostname"=>"so-fn01"}, "message"=>"{\"timestamp\":\"2025-01-22T19:44:32.508492+0000\",\"flow_id\":1561682901267461,\"in_iface\":\"bond0\",\"event_type\":\"ike\",\"src_ip\":\"193.163.125.85\",\"src_port\":22539,\"dest_ip\":\"[a public IPv4 address]\",\"dest_port\":500,\"proto\":\"UDP\",\"pkt_src\":\"wire/pcap\",\"metadata\":{\"flowbits\":[\"ET.Evil\",\"ET.DshieldIP\"]},\"community_id\":\"1:gANtDkBQETLt+46Bvtuvhdxv9/s=\",\"ike\":{\"version_major\":1,\"version_minor\":0,\"init_spi\":\"8937bece547415d3\",\"resp_spi\":\"0000000000000000\",\"message_id\":0,\"exchange_type\":2,\"payload\":[\"Transform\",\"SecurityAssociation\",\"Proposal\",\"VendorID\"],\"ikev1\":{\"doi\":1,\"encrypted_payloads\":false,\"client\":{\"proposals\":[{\"alg_enc\":\"EncAesCbc\",\"alg_enc_raw\":7,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":256,\"alg_hash\":\"HashSha2_384\",\"alg_hash_raw\":5,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupRandomEcp384\",\"alg_dh_raw\":20,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncAesCbc\",\"alg_enc_raw\":7,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":256,\"alg_hash\":\"HashSha2_256\",\"alg_hash_raw\":4,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupModp2048Bit\",\"alg_dh_raw\":14,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncAesCbc\",\"alg_enc_raw\":7,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":128,\"alg_hash\":\"HashSha\",\"alg_hash_raw\":2,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupAlternate1024BitModpGroup\",\"alg_dh_raw\":2,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncTripleDesCbc\",\"alg_enc_raw\":5,\"alg_hash\":\"HashSha\",\"alg_hash_raw\":2,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupAlternate1024BitModpGroup\",\"alg_dh_raw\":2,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncTripleDesCbc\",\"alg_enc_raw\":5,\"alg_hash\":\"HashMd5\",\"alg_hash_raw\":1,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupAlternate1024BitModpGroup\",\"alg_dh_raw\":2,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncDesCbc\",\"alg_enc_raw\":1,\"alg_hash\":\"HashSha\",\"alg_hash_raw\":2,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupDefault768BitModp\",\"alg_dh_raw\":1,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535},{\"alg_enc\":\"EncDesCbc\",\"alg_enc_raw\":1,\"alg_hash\":\"HashMd5\",\"alg_hash_raw\":1,\"alg_auth\":\"AuthPreSharedKey\",\"alg_auth_raw\":1,\"alg_dh\":\"GroupDefault768BitModp\",\"alg_dh_raw\":1,\"sa_life_type\":\"LifeTypeSeconds\",\"sa_life_type_raw\":1,\"sa_life_duration\":\"Unknown\",\"sa_life_duration_raw\":65535}]},\"server\":{},\"vendor_ids\":[\"dc0d196fde592e68fed94cf438429c3c\"]}}}", "event"=>{"module"=>"suricata", "dataset"=>"suricata", "category"=>"network"}, "elastic_agent"=>{"id"=>"e4cbddac-3143-4561-bbab-9afc4a8a235a", "snapshot"=>false, "version"=>"8.14.3"}, "type"=>"redis-input", "log"=>{"offset"=>327571923, "file"=>{"path"=>"/nsm/suricata/eve-2025-01-22-19:01.json"}}, "tags"=>["elastic-agent", "input-so-rn01", "beats_input_codec_plain_applied"], "input"=>{"type"=>"log"}, "agent"=>{"id"=>"e4cbddac-3143-4561-bbab-9afc4a8a235a", "name"=>"so-fn01", "version"=>"8.14.3", "ephemeral_id"=>"76fb3577-1c34-4f2e-b278-403aea88697e", "type"=>"filebeat"}, "@timestamp"=>2025-01-22T19:44:32.539Z}], :error=>{"type"=>"illegal_state_exception", "reason"=>"Pipeline processor configured for non-existent pipeline [suricata.]"}}

[2025-01-22T20:22:27,128][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>1}

I think I got a bit overzealous when I attempted to deal with mismatched indices in Elastic Search. Is there a way of recreating what I deleted, short of reinstalling the whole fleet, or is such a deletion of no concern?

In the meantime, I've created a cron job that restarts the Logstash container on the search node, every hour, on the hour. That seems to keep the wheels turning, but it's a bandaid I would like to remove asap.

[cm-ops]

What pipelines are loaded? sudo so-elasticsearch-pipelines-list The first log looks like an ike log, I have seen this before when the log is malformed json.

What are your mismatched indices? What did you delete?

[Ximalas]

These are the pipelines visible from the manageer:

[
  ".fleet_final_pipeline-1",
  ".slo-observability.sli.pipeline",
  ".slo-observability.sli.pipeline-v3.2",
  ".slo-observability.summary.pipeline",
  "beats.common",
  "behavioral_analytics-events-final_pipeline",
  "common",
  "common.nids",
  "custom001",
  "custom002",
  "custom003",
  "custom004",
  "custom005",
  "custom006",
  "custom007",
  "custom008",
  "custom009",
  "custom010",
  "dns.tld",
  "ecs",
  "ent-search-generic-ingestion",
  "filterlog",
  "global@custom",
  "http.status",
  "import.wel",
  "kismet.ad_hoc",
  "kismet.ap",
  "kismet.bridged",
  "kismet.client",
  "kismet.common",
  "kismet.device",
  "kismet.seenby",
  "kismet.wds",
  "kismet.wds_ap",
  "kratos",
  "logs-1password.audit_events-1.29.0",
  "logs-1password.item_usages-1.29.0",
  "logs-1password.signin_attempts-1.29.0",
  "logs-apache.access-1.21.0",
  "logs-apache.access-1.21.0-third-party",
  "logs-apache.error-1.21.0",
  "logs-apache.error-1.21.0-third-party",
  "logs-auditd.log-3.19.2",
  "logs-auth0.logs-1.16.0",
  "logs-aws.apigateway_logs-2.17.0",
  "logs-aws.cloudfront_logs-2.17.0",
  "logs-aws.cloudtrail-2.17.0",
  "logs-aws.cloudtrail-2.17.0-third-party",
  "logs-aws.cloudwatch_logs-2.17.0",
  "logs-aws.ec2_logs-2.17.0",
  "logs-aws.elb_logs-2.17.0",
  "logs-aws.emr_logs-2.17.0",
  "logs-aws.firewall_logs-2.17.0",
  "logs-aws.guardduty-2.17.0",
  "logs-aws.inspector-2.17.0",
  "logs-aws.route53_public_logs-2.17.0",
  "logs-aws.route53_resolver_logs-2.17.0",
  "logs-aws.s3access-2.17.0",
  "logs-aws.securityhub_findings-2.17.0",
  "logs-aws.securityhub_insights-2.17.0",
  "logs-aws.vpcflow-2.17.0",
  "logs-aws.waf-2.17.0",
  "logs-azure.activitylogs-1.13.0",
  "logs-azure.activitylogs-1.13.0-azure-shared-pipeline",
  "logs-azure.application_gateway-1.13.0",
  "logs-azure.application_gateway-1.13.0-azure-shared-pipeline",
  "logs-azure.auditlogs-1.13.0",
  "logs-azure.auditlogs-1.13.0-azure-shared-pipeline",
  "logs-azure.eventhub-1.13.0",
  "logs-azure.eventhub-1.13.0-azure-shared-pipeline",
  "logs-azure.eventhub-1.13.0-parsed-message",
  "logs-azure.firewall_logs-1.13.0",
  "logs-azure.firewall_logs-1.13.0-azure-shared-pipeline",
  "logs-azure.graphactivitylogs-1.13.0",
  "logs-azure.graphactivitylogs-1.13.0-azure-shared-pipeline",
  "logs-azure.identity_protection-1.13.0",
  "logs-azure.identity_protection-1.13.0-azure-shared-pipeline",
  "logs-azure.platformlogs-1.13.0",
  "logs-azure.platformlogs-1.13.0-azure-shared-pipeline",
  "logs-azure.platformlogs-1.13.0-springcloudlogs-inner-pipeline",
  "logs-azure.provisioning-1.13.0",
  "logs-azure.provisioning-1.13.0-azure-shared-pipeline",
  "logs-azure.signinlogs-1.13.0",
  "logs-azure.signinlogs-1.13.0-azure-shared-pipeline",
  "logs-azure.springcloudlogs-1.13.0",
  "logs-azure.springcloudlogs-1.13.0-azure-shared-pipeline",
  "logs-barracuda.waf-1.14.0",
  "logs-barracuda.waf-1.14.0-access",
  "logs-barracuda.waf-1.14.0-audit",
  "logs-barracuda.waf-1.14.0-networkfirewall",
  "logs-barracuda.waf-1.14.0-system",
  "logs-barracuda.waf-1.14.0-webfirewall",
  "logs-barracuda_cloudgen_firewall.log-1.12.0",
  "logs-barracuda_cloudgen_firewall.log-1.12.0-firewall",
  "logs-barracuda_cloudgen_firewall.log-1.12.0-threat",
  "logs-barracuda_cloudgen_firewall.log-1.12.0-web",
  "logs-carbonblack_edr.log-1.18.0",
  "logs-cef.log-2.17.1",
  "logs-cef.log-2.17.1-cp-pipeline",
  "logs-cef.log-2.17.1-fp-pipeline",
  "logs-checkpoint.firewall-1.31.0",
  "logs-cisco_asa.log-2.36.0",
  "logs-cisco_duo.admin-1.25.0",
  "logs-cisco_duo.auth-1.25.0",
  "logs-cisco_duo.offline_enrollment-1.25.0",
  "logs-cisco_duo.summary-1.25.0",
  "logs-cisco_duo.telephony-1.25.0",
  "logs-cisco_ftd.log-3.3.0",
  "logs-cisco_ios.log-1.26.10",
  "logs-cisco_ise.log-1.22.1",
  "logs-cisco_ise.log-1.22.1-pipeline_ad_connector",
  "logs-cisco_ise.log-1.22.1-pipeline_administrative_and_operational_audit",
  "logs-cisco_ise.log-1.22.1-pipeline_alarm",
  "logs-cisco_ise.log-1.22.1-pipeline_authentication_flow_diagnostics",
  "logs-cisco_ise.log-1.22.1-pipeline_failed_attempts",
  "logs-cisco_ise.log-1.22.1-pipeline_guest",
  "logs-cisco_ise.log-1.22.1-pipeline_identity_stores_diagnostics",
  "logs-cisco_ise.log-1.22.1-pipeline_internal_operations_diagnostics",
  "logs-cisco_ise.log-1.22.1-pipeline_mydevices",
  "logs-cisco_ise.log-1.22.1-pipeline_passed_authentications",
  "logs-cisco_ise.log-1.22.1-pipeline_policy_diagnostics",
  "logs-cisco_ise.log-1.22.1-pipeline_posture_and_client_provisioning_audit",
  "logs-cisco_ise.log-1.22.1-pipeline_radius_accounting",
  "logs-cisco_ise.log-1.22.1-pipeline_radius_diagnostics",
  "logs-cisco_ise.log-1.22.1-pipeline_system_statistics",
  "logs-cisco_ise.log-1.22.1-pipeline_tacacs_accounting",
  "logs-cisco_ise.log-1.22.1-pipeline_threat_centric_nac",
  "logs-cisco_meraki.events-1.23.0",
  "logs-cisco_meraki.log-1.23.0",
  "logs-cisco_meraki.log-1.23.0-airmarshal",
  "logs-cisco_meraki.log-1.23.0-events",
  "logs-cisco_meraki.log-1.23.0-flows",
  "logs-cisco_meraki.log-1.23.0-idsalerts",
  "logs-cisco_meraki.log-1.23.0-ipflows",
  "logs-cisco_meraki.log-1.23.0-security",
  "logs-cisco_meraki.log-1.23.0-urls",
  "logs-cisco_umbrella.log-1.25.0",
  "logs-citrix_adc.interface-1.7.0",
  "logs-citrix_adc.lbvserver-1.7.0",
  "logs-citrix_adc.log-1.7.0",
  "logs-citrix_adc.log-1.7.0-alg_feature",
  "logs-citrix_adc.log-1.7.0-appfw_feature",
  "logs-citrix_adc.log-1.7.0-bot_feature",
  "logs-citrix_adc.log-1.7.0-cef",
  "logs-citrix_adc.log-1.7.0-ci_feature",
  "logs-citrix_adc.log-1.7.0-cvpn_feature",
  "logs-citrix_adc.log-1.7.0-dns_and_ssli_feature",
  "logs-citrix_adc.log-1.7.0-ica_feature",
  "logs-citrix_adc.log-1.7.0-native",
  "logs-citrix_adc.log-1.7.0-pitboss_feature",
  "logs-citrix_adc.log-1.7.0-ssllog_feature",
  "logs-citrix_adc.log-1.7.0-sslvpn_and_aaatm_feature",
  "logs-citrix_adc.log-1.7.0-tcp_and_acl_feature",
  "logs-citrix_adc.log-1.7.0-transform_feature",
  "logs-citrix_adc.service-1.7.0",
  "logs-citrix_adc.system-1.7.0",
  "logs-citrix_adc.vpn-1.7.0",
  "logs-citrix_waf.log-1.15.0",
  "logs-citrix_waf.log-1.15.0-cef",
  "logs-citrix_waf.log-1.15.0-native",
  "logs-cloudflare.audit-2.27.0",
  "logs-cloudflare.logpull-2.27.0",
  "logs-cloudflare.logpull-2.27.0-http",
  "logs-crowdstrike.alert-1.38.0",
  "logs-crowdstrike.falcon-1.38.0",
  "logs-crowdstrike.falcon-1.38.0-auth_activity_audit",
  "logs-crowdstrike.falcon-1.38.0-cspm_events",
  "logs-crowdstrike.falcon-1.38.0-detection_summary",
  "logs-crowdstrike.falcon-1.38.0-firewall_match",
  "logs-crowdstrike.falcon-1.38.0-identity_protection_incident",
  "logs-crowdstrike.falcon-1.38.0-incident_summary",
  "logs-crowdstrike.falcon-1.38.0-ipd_detection_summary",
  "logs-crowdstrike.falcon-1.38.0-mobile_detection_summary",
  "logs-crowdstrike.falcon-1.38.0-recon_notification_summary",
  "logs-crowdstrike.falcon-1.38.0-remote_response_session_end",
  "logs-crowdstrike.falcon-1.38.0-remote_response_session_start",
  "logs-crowdstrike.falcon-1.38.0-scheduled_report_notification_event",
  "logs-crowdstrike.falcon-1.38.0-user_activity_audit",
  "logs-crowdstrike.falcon-1.38.0-xdr_detection_summary",
  "logs-crowdstrike.fdr-1.38.0",
  "logs-crowdstrike.host-1.38.0",
  "logs-darktrace.ai_analyst_alert-1.18.0",
  "logs-darktrace.model_breach_alert-1.18.0",
  "logs-darktrace.system_status_alert-1.18.0",
  "logs-default-pipeline",
  "logs-elastic_agent-1.13.1",
  "logs-elastic_agent-1.20.0",
  "logs-elastic_agent.apm_server-1.20.0",
  "logs-elastic_agent.auditbeat-1.20.0",
  "logs-elastic_agent.cloud_defend-1.20.0",
  "logs-elastic_agent.cloudbeat-1.20.0",
  "logs-elastic_agent.endpoint_security-1.20.0",
  "logs-elastic_agent.filebeat-1.13.1",
  "logs-elastic_agent.filebeat-1.20.0",
  "logs-elastic_agent.filebeat_input-1.20.0",
  "logs-elastic_agent.fleet_server-1.13.1",
  "logs-elastic_agent.fleet_server-1.20.0",
  "logs-elastic_agent.heartbeat-1.20.0",
  "logs-elastic_agent.metricbeat-1.20.0",
  "logs-elastic_agent.osquerybeat-1.13.1",
  "logs-elastic_agent.osquerybeat-1.20.0",
  "logs-elastic_agent.packetbeat-1.20.0",
  "logs-elastic_agent.pf_elastic_collector-1.20.0",
  "logs-elastic_agent.pf_elastic_symbolizer-1.20.0",
  "logs-elastic_agent.pf_host_agent-1.20.0",
  "logs-elasticsearch.audit-1.15.0",
  "logs-elasticsearch.audit-1.15.0-pipeline-json",
  "logs-elasticsearch.deprecation-1.15.0",
  "logs-elasticsearch.deprecation-1.15.0-pipeline-json",
  "logs-elasticsearch.gc-1.15.0",
  "logs-elasticsearch.server-1.15.0",
  "logs-elasticsearch.server-1.15.0-pipeline-json",
  "logs-elasticsearch.slowlog-1.15.0",
  "logs-elasticsearch.slowlog-1.15.0-pipeline-json",
  "logs-endpoint.action.responses-8.14.0",
  "logs-endpoint.actions-8.14.0",
  "logs-endpoint.alerts-8.14.0",
  "logs-endpoint.diagnostic.collection-8.10.2",
  "logs-endpoint.diagnostic.collection-8.14.0",
  "logs-endpoint.events.api-8.14.0",
  "logs-endpoint.events.file-8.14.0",
  "logs-endpoint.events.library-8.14.0",
  "logs-endpoint.events.network-8.14.0",
  "logs-endpoint.events.process-8.14.0",
  "logs-endpoint.events.registry-8.14.0",
  "logs-endpoint.events.security-8.14.0",
  "logs-endpoint.heartbeat-8.14.0",
  "logs-f5_bigip.log-1.17.0",
  "logs-f5_bigip.log-1.17.0-pipeline_bigipafm",
  "logs-f5_bigip.log-1.17.0-pipeline_bigipapm",
  "logs-f5_bigip.log-1.17.0-pipeline_bigipasm",
  "logs-f5_bigip.log-1.17.0-pipeline_bigipavr",
  "logs-f5_bigip.log-1.17.0-pipeline_bigipltm",
  "logs-fim.event-1.15.1",
  "logs-fireeye.nx-1.23.0",
  "logs-fireeye.nx-1.23.0-renaming-raws",
  "logs-fleet_server.output_health-1.5.0",
  "logs-fortinet.clientendpoint-1.9.0",
  "logs-fortinet.firewall-1.9.0",
  "logs-fortinet.firewall-1.9.0-event",
  "logs-fortinet.firewall-1.9.0-traffic",
  "logs-fortinet.firewall-1.9.0-utm",
  "logs-fortinet.fortimail-1.9.0",
  "logs-fortinet.fortimanager-1.9.0",
  "logs-fortinet_fortigate.log-1.25.4",
  "logs-fortinet_fortigate.log-1.25.4-event",
  "logs-fortinet_fortigate.log-1.25.4-login",
  "logs-fortinet_fortigate.log-1.25.4-traffic",
  "logs-fortinet_fortigate.log-1.25.4-utm",
  "logs-gcp.audit-2.35.0",
  "logs-gcp.dns-2.35.0",
  "logs-gcp.firewall-2.35.0",
  "logs-gcp.loadbalancing_logs-2.35.0",
  "logs-gcp.vpcflow-2.35.0",
  "logs-github.audit-1.29.0",
  "logs-github.code_scanning-1.29.0",
  "logs-github.dependabot-1.29.0",
  "logs-github.issues-1.29.0",
  "logs-github.secret_scanning-1.29.0",
  "logs-google_workspace.access_transparency-2.23.0",
  "logs-google_workspace.admin-2.23.0",
  "logs-google_workspace.alert-2.23.0",
  "logs-google_workspace.context_aware_access-2.23.0",
  "logs-google_workspace.device-2.23.0",
  "logs-google_workspace.drive-2.23.0",
  "logs-google_workspace.gcp-2.23.0",
  "logs-google_workspace.group_enterprise-2.23.0",
  "logs-google_workspace.groups-2.23.0",
  "logs-google_workspace.login-2.23.0",
  "logs-google_workspace.rules-2.23.0",
  "logs-google_workspace.saml-2.23.0",
  "logs-google_workspace.token-2.23.0",
  "logs-google_workspace.user_accounts-2.23.0",
  "logs-http_endpoint.generic-2.2.1",
  "logs-httpjson.generic-1.21.0",
  "logs-idh-2.3.1",
  "logs-iis.access-1.20.0",
  "logs-iis.error-1.20.0",
  "logs-imperva_cloud_waf.event-1.1.1",
  "logs-import-2.3.1",
  "logs-juniper.junos-1.2.0",
  "logs-juniper.netscreen-1.2.0",
  "logs-juniper.srx-1.2.0",
  "logs-juniper.srx-1.2.0-atp",
  "logs-juniper.srx-1.2.0-flow",
  "logs-juniper.srx-1.2.0-idp",
  "logs-juniper.srx-1.2.0-ids",
  "logs-juniper.srx-1.2.0-secintel",
  "logs-juniper.srx-1.2.0-utm",
  "logs-juniper_srx.log-1.21.0",
  "logs-juniper_srx.log-1.21.0-atp",
  "logs-juniper_srx.log-1.21.0-flow",
  "logs-juniper_srx.log-1.21.0-idp",
  "logs-juniper_srx.log-1.21.0-ids",
  "logs-juniper_srx.log-1.21.0-secintel",
  "logs-juniper_srx.log-1.21.0-system",
  "logs-juniper_srx.log-1.21.0-utm",
  "logs-kafka_log.generic-1.7.0",
  "logs-kratos-2.3.1",
  "logs-lastpass.detailed_shared_folder-1.17.0",
  "logs-lastpass.event_report-1.17.0",
  "logs-lastpass.user-1.17.0",
  "logs-m365_defender.alert-2.14.1",
  "logs-m365_defender.event-2.14.1",
  "logs-m365_defender.event-2.14.1-pipeline_alert",
  "logs-m365_defender.event-2.14.1-pipeline_app_and_identity",
  "logs-m365_defender.event-2.14.1-pipeline_device",
  "logs-m365_defender.event-2.14.1-pipeline_email",
  "logs-m365_defender.incident-2.14.1",
  "logs-m365_defender.log-2.14.1",
  "logs-microsoft_defender_endpoint.log-2.25.0",
  "logs-microsoft_dhcp.log-1.24.2",
  "logs-microsoft_dhcp.log-1.24.2-dhcp",
  "logs-microsoft_dhcp.log-1.24.2-dhcpv6",
  "logs-microsoft_sqlserver.audit-2.7.0",
  "logs-microsoft_sqlserver.log-2.7.0",
  "logs-mimecast.archive_search_logs-1.26.1",
  "logs-mimecast.audit_events-1.26.1",
  "logs-mimecast.dlp_logs-1.26.1",
  "logs-mimecast.siem_logs-1.26.1",
  "logs-mimecast.threat_intel_malware_customer-1.26.1",
  "logs-mimecast.threat_intel_malware_grid-1.26.1",
  "logs-mimecast.ttp_ap_logs-1.26.1",
  "logs-mimecast.ttp_ip_logs-1.26.1",
  "logs-mimecast.ttp_url_logs-1.26.1",
  "logs-mysql.error-1.22.0",
  "logs-mysql.slowlog-1.22.0",
  "logs-netflow.log-2.18.0",
  "logs-nginx.access-1.22.0",
  "logs-nginx.access-1.22.0-third-party",
  "logs-nginx.error-1.22.0",
  "logs-nginx.error-1.22.0-third-party",
  "logs-o365.audit-2.5.1",
  "logs-okta.system-2.11.0",
  "logs-osquery_manager.result-1.11.0",
  "logs-panw.panos-3.26.2",
  "logs-panw.panos-3.26.2-audit",
  "logs-panw.panos-3.26.2-authentication",
  "logs-panw.panos-3.26.2-config",
  "logs-panw.panos-3.26.2-correlated_event",
  "logs-panw.panos-3.26.2-decryption",
  "logs-panw.panos-3.26.2-globalprotect",
  "logs-panw.panos-3.26.2-gtp",
  "logs-panw.panos-3.26.2-hipmatch",
  "logs-panw.panos-3.26.2-ip_tag",
  "logs-panw.panos-3.26.2-sctp",
  "logs-panw.panos-3.26.2-system",
  "logs-panw.panos-3.26.2-threat",
  "logs-panw.panos-3.26.2-traffic",
  "logs-panw.panos-3.26.2-tunnel_inspection",
  "logs-panw.panos-3.26.2-userid",
  "logs-pfsense.log-1.16.0",
  "logs-pfsense.log-1.16.0-suricata",
  "logs-pfsense.log-1.19.1",
  "logs-pfsense.log-1.19.1-dhcp",
  "logs-pfsense.log-1.19.1-firewall",
  "logs-pfsense.log-1.19.1-haproxy",
  "logs-pfsense.log-1.19.1-ipsec",
  "logs-pfsense.log-1.19.1-openvpn",
  "logs-pfsense.log-1.19.1-php-fpm",
  "logs-pfsense.log-1.19.1-squid",
  "logs-pfsense.log-1.19.1-unbound",
  "logs-proofpoint_tap.clicks_blocked-1.22.0",
  "logs-proofpoint_tap.clicks_permitted-1.22.0",
  "logs-proofpoint_tap.message_blocked-1.22.0",
  "logs-proofpoint_tap.message_delivered-1.22.0",
  "logs-pulse_connect_secure.log-2.1.0",
  "logs-redis.log-1.17.0",
  "logs-redis.slowlog-1.17.0",
  "logs-rita-2.3.1",
  "logs-sentinel_one.activity-1.25.0",
  "logs-sentinel_one.agent-1.25.0",
  "logs-sentinel_one.alert-1.25.0",
  "logs-sentinel_one.group-1.25.0",
  "logs-sentinel_one.threat-1.25.0",
  "logs-snort.log-1.15.0",
  "logs-snort.log-1.15.0-json",
  "logs-snort.log-1.15.0-plaintext",
  "logs-snyk.audit-1.24.0",
  "logs-snyk.audit_logs-1.24.0",
  "logs-snyk.issues-1.24.0",
  "logs-snyk.vulnerabilities-1.24.0",
  "logs-soc-2.3.1",
  "logs-sonicwall_firewall.log-1.16.0",
  "logs-sophos.utm-3.9.0",
  "logs-sophos.utm-3.9.0-dhcp",
  "logs-sophos.utm-3.9.0-dns",
  "logs-sophos.utm-3.9.0-http",
  "logs-sophos.utm-3.9.0-packetfilter",
  "logs-sophos.xg-3.9.0",
  "logs-sophos.xg-3.9.0-antispam",
  "logs-sophos.xg-3.9.0-antivirus",
  "logs-sophos.xg-3.9.0-atp",
  "logs-sophos.xg-3.9.0-cfilter",
  "logs-sophos.xg-3.9.0-event",
  "logs-sophos.xg-3.9.0-firewall",
  "logs-sophos.xg-3.9.0-idp",
  "logs-sophos.xg-3.9.0-sandstorm",
  "logs-sophos.xg-3.9.0-systemhealth",
  "logs-sophos.xg-3.9.0-waf",
  "logs-sophos.xg-3.9.0-wifi",
  "logs-sophos_central.alert-1.15.0",
  "logs-sophos_central.event-1.15.0",
  "logs-strelka-2.3.1",
  "logs-suricata-2.3.1",
  "logs-symantec_endpoint.log-2.16.0",
  "logs-system.application-1.59.0",
  "logs-system.auth-1.43.0",
  "logs-system.auth-1.59.0",
  "logs-system.security-1.59.0",
  "logs-system.security-1.59.0-standard",
  "logs-system.syslog-1.43.0",
  "logs-system.syslog-1.59.0",
  "logs-system.system-1.59.0",
  "logs-tcp.generic-1.19.0",
  "logs-tenable_io.asset-3.2.0",
  "logs-tenable_io.plugin-3.2.0",
  "logs-tenable_io.scan-3.2.0",
  "logs-tenable_io.vulnerability-3.2.0",
  "logs-tenable_sc.asset-1.23.0",
  "logs-tenable_sc.plugin-1.23.0",
  "logs-tenable_sc.vulnerability-1.23.0",
  "logs-ti_abusech.malware-2.3.0",
  "logs-ti_abusech.malwarebazaar-2.3.0",
  "logs-ti_abusech.threatfox-2.3.0",
  "logs-ti_abusech.url-2.3.0",
  "logs-ti_anomali.threatstream-1.22.0",
  "logs-ti_cybersixgill.threat-1.30.0",
  "logs-ti_misp.threat-1.35.0",
  "logs-ti_misp.threat_attributes-1.35.0",
  "logs-ti_otx.pulses_subscribed-1.25.0",
  "logs-ti_otx.threat-1.25.0",
  "logs-ti_recordedfuture.threat-1.26.0",
  "logs-ti_recordedfuture.threat-1.26.0-decode_csv",
  "logs-ti_threatq.threat-1.28.0",
  "logs-udp.generic-1.19.0",
  "logs-vsphere.log-1.13.0",
  "logs-vsphere.log-1.13.0-dns",
  "logs-vsphere.log-1.13.0-file",
  "logs-vsphere.log-1.13.0-login",
  "logs-windows.applocker_exe_and_dll-1.45.1",
  "logs-windows.applocker_msi_and_script-1.45.1",
  "logs-windows.applocker_packaged_app_deployment-1.45.1",
  "logs-windows.applocker_packaged_app_execution-1.45.1",
  "logs-windows.forwarded-1.45.1",
  "logs-windows.forwarded-1.45.1-powershell",
  "logs-windows.forwarded-1.45.1-powershell_operational",
  "logs-windows.forwarded-1.45.1-security",
  "logs-windows.forwarded-1.45.1-sysmon_operational",
  "logs-windows.powershell-1.45.1",
  "logs-windows.powershell_operational-1.45.1",
  "logs-windows.sysmon_operational-1.45.1",
  "logs-winlog.winlog-2.1.2",
  "logs-zeek-2.3.1",
  "logs-zscaler_zia.alerts-3.0.0",
  "logs-zscaler_zia.audit-3.0.0",
  "logs-zscaler_zia.dns-3.0.0",
  "logs-zscaler_zia.endpoint_dlp-3.0.0",
  "logs-zscaler_zia.firewall-3.0.0",
  "logs-zscaler_zia.sandbox_report-3.0.0",
  "logs-zscaler_zia.tunnel-3.0.0",
  "logs-zscaler_zia.web-3.0.0",
  "logs-zscaler_zpa.app_connector_status-1.18.0",
  "logs-zscaler_zpa.audit-1.18.0",
  "logs-zscaler_zpa.browser_access-1.18.0",
  "logs-zscaler_zpa.user_activity-1.18.0",
  "logs-zscaler_zpa.user_status-1.18.0",
  "logs@default-pipeline",
  "logs@json-message",
  "logs@json-pipeline",
  "logscan.alert",
  "metrics-apache.status-1.21.0",
  "metrics-aws.apigateway_metrics-2.17.0",
  "metrics-aws.billing-2.17.0",
  "metrics-aws.cloudwatch_metrics-2.17.0",
  "metrics-aws.dynamodb-2.17.0",
  "metrics-aws.ebs-2.17.0",
  "metrics-aws.ec2_metrics-2.17.0",
  "metrics-aws.ecs_metrics-2.17.0",
  "metrics-aws.elb_metrics-2.17.0",
  "metrics-aws.emr_metrics-2.17.0",
  "metrics-aws.firewall_metrics-2.17.0",
  "metrics-aws.kafka_metrics-2.17.0",
  "metrics-aws.kinesis-2.17.0",
  "metrics-aws.lambda-2.17.0",
  "metrics-aws.natgateway-2.17.0",
  "metrics-aws.rds-2.17.0",
  "metrics-aws.redshift-2.17.0",
  "metrics-aws.s3_daily_storage-2.17.0",
  "metrics-aws.s3_request-2.17.0",
  "metrics-aws.s3_storage_lens-2.17.0",
  "metrics-aws.sns-2.17.0",
  "metrics-aws.sqs-2.17.0",
  "metrics-aws.transitgateway-2.17.0",
  "metrics-aws.usage-2.17.0",
  "metrics-aws.vpn-2.17.0",
  "metrics-elastic_agent.apm_server-1.20.0",
  "metrics-elastic_agent.auditbeat-1.20.0",
  "metrics-elastic_agent.cloudbeat-1.20.0",
  "metrics-elastic_agent.elastic_agent-1.20.0",
  "metrics-elastic_agent.endpoint_security-1.20.0",
  "metrics-elastic_agent.filebeat-1.20.0",
  "metrics-elastic_agent.filebeat_input-1.20.0",
  "metrics-elastic_agent.fleet_server-1.20.0",
  "metrics-elastic_agent.heartbeat-1.20.0",
  "metrics-elastic_agent.metricbeat-1.20.0",
  "metrics-elastic_agent.osquerybeat-1.20.0",
  "metrics-elastic_agent.packetbeat-1.20.0",
  "metrics-elasticsearch.ingest_pipeline-1.15.0",
  "metrics-elasticsearch.stack_monitoring.ccr-1.15.0",
  "metrics-elasticsearch.stack_monitoring.cluster_stats-1.15.0",
  "metrics-elasticsearch.stack_monitoring.enrich-1.15.0",
  "metrics-elasticsearch.stack_monitoring.index-1.15.0",
  "metrics-elasticsearch.stack_monitoring.index_recovery-1.15.0",
  "metrics-elasticsearch.stack_monitoring.index_summary-1.15.0",
  "metrics-elasticsearch.stack_monitoring.ml_job-1.15.0",
  "metrics-elasticsearch.stack_monitoring.node-1.15.0",
  "metrics-elasticsearch.stack_monitoring.node_stats-1.15.0",
  "metrics-elasticsearch.stack_monitoring.pending_tasks-1.15.0",
  "metrics-elasticsearch.stack_monitoring.shard-1.15.0",
  "metrics-endpoint.metadata-8.14.0",
  "metrics-endpoint.metrics-8.14.0",
  "metrics-endpoint.policy-8.14.0",
  "metrics-fleet_server.agent_status-1.5.0",
  "metrics-fleet_server.agent_versions-1.5.0",
  "metrics-gcp.billing-2.35.0",
  "metrics-gcp.cloudrun_metrics-2.35.0",
  "metrics-gcp.cloudsql_mysql-2.35.0",
  "metrics-gcp.cloudsql_postgresql-2.35.0",
  "metrics-gcp.cloudsql_sqlserver-2.35.0",
  "metrics-gcp.compute-2.35.0",
  "metrics-gcp.dataproc-2.35.0",
  "metrics-gcp.firestore-2.35.0",
  "metrics-gcp.gke-2.35.0",
  "metrics-gcp.loadbalancing_metrics-2.35.0",
  "metrics-gcp.pubsub-2.35.0",
  "metrics-gcp.redis-2.35.0",
  "metrics-gcp.storage-2.35.0",
  "metrics-iis.application_pool-1.20.0",
  "metrics-iis.webserver-1.20.0",
  "metrics-iis.website-1.20.0",
  "metrics-microsoft_sqlserver.performance-2.7.0",
  "metrics-microsoft_sqlserver.transaction_log-2.7.0",
  "metrics-mysql.galera_status-1.22.0",
  "metrics-mysql.performance-1.22.0",
  "metrics-mysql.status-1.22.0",
  "metrics-nginx.stubstatus-1.22.0",
  "metrics-redis.info-1.17.0",
  "metrics-redis.key-1.17.0",
  "metrics-redis.keyspace-1.17.0",
  "metrics-system.core-1.59.0",
  "metrics-system.cpu-1.59.0",
  "metrics-system.diskio-1.59.0",
  "metrics-system.filesystem-1.59.0",
  "metrics-system.fsstat-1.59.0",
  "metrics-system.load-1.59.0",
  "metrics-system.memory-1.59.0",
  "metrics-system.network-1.59.0",
  "metrics-system.process-1.59.0",
  "metrics-system.process.summary-1.59.0",
  "metrics-system.socket_summary-1.59.0",
  "metrics-system.uptime-1.59.0",
  "metrics-vsphere.datastore-1.13.0",
  "metrics-vsphere.host-1.13.0",
  "metrics-vsphere.virtualmachine-1.13.0",
  "metrics-windows.perfmon-1.45.1",
  "metrics-windows.service-1.45.1",
  "osquery.live_query",
  "osquery.normalize",
  "osquery.query_result",
  "ossec",
  "rita.beacon",
  "rita.beacons",
  "rita.connection",
  "rita.connections",
  "rita.dns",
  "strelka.file",
  "sublime",
  "suricata.alert",
  "suricata.alert_pfsense",
  "suricata.common",
  "suricata.common_pfsense",
  "suricata.dhcp",
  "suricata.dnp3",
  "suricata.dns",
  "suricata.fileinfo",
  "suricata.flow",
  "suricata.ftp",
  "suricata.ftp_data",
  "suricata.http",
  "suricata.ike",
  "suricata.ikev2",
  "suricata.krb5",
  "suricata.nfs",
  "suricata.rdp",
  "suricata.sip",
  "suricata.smb",
  "suricata.smtp",
  "suricata.snmp",
  "suricata.ssh",
  "suricata.tftp",
  "suricata.tls",
  "synthetics-browser-1.2.1",
  "synthetics-browser.network-1.2.1",
  "synthetics-browser.screenshot-1.2.1",
  "synthetics-http-1.2.1",
  "synthetics-icmp-1.2.1",
  "synthetics-tcp-1.2.1",
  "syslog",
  "sysmon",
  "win.eventlogs",
  "zeek.bacnet",
  "zeek.bacnet_discovery",
  "zeek.bacnet_property",
  "zeek.bsap_ip_header",
  "zeek.bsap_ip_rdb",
  "zeek.bsap_ip_unknown",
  "zeek.bsap_serial_header",
  "zeek.bsap_serial_rdb",
  "zeek.bsap_serial_rdb_ext",
  "zeek.bsap_serial_unknown",
  "zeek.cip",
  "zeek.cip_identity",
  "zeek.cip_io",
  "zeek.common",
  "zeek.common_ssl",
  "zeek.conn",
  "zeek.cotp",
  "zeek.dce_rpc",
  "zeek.dhcp",
  "zeek.dnp3",
  "zeek.dnp3_control",
  "zeek.dnp3_objects",
  "zeek.dns",
  "zeek.dpd",
  "zeek.ecat_aoe_info",
  "zeek.ecat_arp_info",
  "zeek.ecat_coe_info",
  "zeek.ecat_dev_info",
  "zeek.ecat_foe_info",
  "zeek.ecat_log_address",
  "zeek.ecat_registers",
  "zeek.ecat_soe_info",
  "zeek.enip",
  "zeek.files",
  "zeek.ftp",
  "zeek.http",
  "zeek.intel",
  "zeek.irc",
  "zeek.kerberos",
  "zeek.modbus",
  "zeek.modbus_detailed",
  "zeek.modbus_mask_write_register",
  "zeek.modbus_read_write_multiple_registers",
  "zeek.mysql",
  "zeek.notice",
  "zeek.ntlm",
  "zeek.opcua_binary",
  "zeek.opcua_binary_activate_session",
  "zeek.opcua_binary_activate_session_client_software_cert",
  "zeek.opcua_binary_activate_session_diagnostic_info",
  "zeek.opcua_binary_activate_session_locale_id",
  "zeek.opcua_binary_browse",
  "zeek.opcua_binary_browse_description",
  "zeek.opcua_binary_browse_diagnostic_info",
  "zeek.opcua_binary_browse_request_continuation_point",
  "zeek.opcua_binary_browse_response_references",
  "zeek.opcua_binary_browse_result",
  "zeek.opcua_binary_create_session",
  "zeek.opcua_binary_create_session_discovery",
  "zeek.opcua_binary_create_session_endpoints",
  "zeek.opcua_binary_create_session_user_token",
  "zeek.opcua_binary_create_subscription",
  "zeek.opcua_binary_diag_info_detail",
  "zeek.opcua_binary_get_endpoints",
  "zeek.opcua_binary_get_endpoints_description",
  "zeek.opcua_binary_get_endpoints_discovery",
  "zeek.opcua_binary_get_endpoints_locale_id",
  "zeek.opcua_binary_get_endpoints_profile_uri",
  "zeek.opcua_binary_get_endpoints_user_token",
  "zeek.opcua_binary_opensecure_channel",
  "zeek.opcua_binary_read",
  "zeek.opcua_binary_read_array_dims",
  "zeek.opcua_binary_read_array_dims_link",
  "zeek.opcua_binary_read_diagnostic_info",
  "zeek.opcua_binary_read_extension_object",
  "zeek.opcua_binary_read_extension_object_link",
  "zeek.opcua_binary_read_nodes_to_read",
  "zeek.opcua_binary_read_results",
  "zeek.opcua_binary_read_results_link",
  "zeek.opcua_binary_read_status_code",
  "zeek.opcua_binary_read_variant_data",
  "zeek.opcua_binary_read_variant_data_link",
  "zeek.opcua_binary_status_code_detail",
  "zeek.pe",
  "zeek.profinet",
  "zeek.profinet_dce_rpc",
  "zeek.radius",
  "zeek.rdp",
  "zeek.rfb",
  "zeek.s7comm",
  "zeek.s7comm_plus",
  "zeek.s7comm_read_szl",
  "zeek.s7comm_upload_download",
  "zeek.signatures",
  "zeek.sip",
  "zeek.smb_files",
  "zeek.smb_mapping",
  "zeek.smtp",
  "zeek.snmp",
  "zeek.socks",
  "zeek.software",
  "zeek.ssh",
  "zeek.ssl",
  "zeek.stun",
  "zeek.stun_nat",
  "zeek.syslog",
  "zeek.tds",
  "zeek.tds_rpc",
  "zeek.tds_sql_batch",
  "zeek.tunnel",
  "zeek.tunnels",
  "zeek.weird",
  "zeek.wireguard",
  "zeek.x509"
]

Instead of limiting the deletion to those indices affected, I'm afraid I deleted everything. Instead of specifying each index individually, I adapted this example from the release notes. Next time, I'll lock my office and be extra careful.

for i in logs-system.security logs-system.syslog; do
    INDEX_TO_DELETE=$(so-elasticsearch-query $i | jq -r 'keys[]' | tail -2 | head -1); so-elasticsearch-query $INDEX_TO_DELETE -XDELETE
done

I'm afraid the commands I issued have been lost. For the future, I added this file to the manager, ~/.bashrc.d/longer-history:

HISTSIZE=10000000

[cm-ops]

Looks like all the pipelines are loaded. You could check suricata.common pipeline stats to see if something is failing on one of the processors. sudo so-elasticsearch-pipeline-stats suricata.common. That will work to troubleshoot any Elasticsearch ingest pipelines above.

[Ximalas]

I noticed this snippet:

    {
      "pipeline:suricata.": {
        "type": "pipeline",
        "stats": {
          "count": 1821372,
          "time_in_millis": 566457,
          "current": 0,
          "failed": 695
        }
      }
    }

Here's the output in its entirety:

{
  "count": 0,
  "time_in_millis": 0,
  "current": 0,
  "failed": 0,
  "processors": [
    {
      "json": {
        "type": "json",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "date": {
        "type": "date",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "remove": {
        "type": "remove",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "pipeline:suricata.": {
        "type": "pipeline",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    }
  ]
}
{
  "count": 1821372,
  "time_in_millis": 807871,
  "current": 0,
  "failed": 695,
  "processors": [
    {
      "json": {
        "type": "json",
        "stats": {
          "count": 1821372,
          "time_in_millis": 44765,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 20559,
          "current": 0,
          "failed": 1066904
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 9303,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 7970,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 7188,
          "current": 0,
          "failed": 1032
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 6024,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 5876,
          "current": 0,
          "failed": 26693
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 5636,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 6147,
          "current": 0,
          "failed": 26693
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 18831,
          "current": 0,
          "failed": 1821372
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 8384,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "rename": {
        "type": "rename",
        "stats": {
          "count": 1821372,
          "time_in_millis": 1249,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 1821372,
          "time_in_millis": 13673,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 1821372,
          "time_in_millis": 7007,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "set": {
        "type": "set",
        "stats": {
          "count": 1821372,
          "time_in_millis": 4583,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "date": {
        "type": "date",
        "stats": {
          "count": 1821372,
          "time_in_millis": 40948,
          "current": 0,
          "failed": 695
        }
      }
    },
    {
      "remove": {
        "type": "remove",
        "stats": {
          "count": 1821372,
          "time_in_millis": 2248,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "pipeline:suricata.": {
        "type": "pipeline",
        "stats": {
          "count": 1821372,
          "time_in_millis": 566457,
          "current": 0,
          "failed": 695
        }
      }
    }
  ]
}