FIX: outdated import-evtx-logs pipeline versions

[chateaulav]

• duplicated the issue on a fresh installation of the latest version

information about your system and how you installed Security Onion

Oracle Linux Server release 9.3
Linux version 5.15.0-101.103.2.1.el9uek.x86_64 (mockbuild@host-100-100-224-7) (gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2.1.0.2), GNU ld version 2.35.2-24.0.1.el9) #2 SMP Tue May 2 01:10:45 PDT 2023

Security Onion Version: 2.4.30

relevant log files

the import is trying to use logs-system.security-1.34.0 based on the elastic-agent policy when Elastic and Logstash are configured with logs-system.security-1.43.0

This applies to the following pipelines, based on the preset agent policy that was not updated with the 2.4 release.

logs-system.system-1.34.0 -> logs-system.system-1.43.0
logs-system.security-1.34.0 -> logs-system.security-1.43.0
logs-system.application-1.34.0 -> logs-system.application-1.43.0
logs-windows.sysmon_operational-1.24.0 -> logs-windows.sysmon_operational-1.38.0
logs-windows.powershell_operational-1.24.0 -> logs-windows.powershell_operational-1.38.0

Affected file:
salt/elasticfleet/files/integrations/grid-nodes_general/import-evtx-logs.json

Logstash ingest pipeline versions:

Logstash error when attempting to import evtx data:

=========================================================================
 Checking log file /opt/so/log/logstash/logstash.log
=========================================================================
[2023-11-29T12:44:31,487][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-import-so", :routing=>nil, :pipeline=>"logs-windows.sysmon_operational-1.24.0"}, {"log"=>{"offset"=>242913, "file"=>{"path"=>"/nsm/import/fff4261b8064add1ddbb546f4c59dece/evtx/data.json", "name"=>"/tmp/data.evtx"}}, "event"=>{"dataset"=>"windows.sysmon_operational", "created"=>"2023-11-29T07:48:43.020873Z", "imported"=>true, "module"=>"windows", "code"=>1}, "message"=>"{\"event_record_id\":13741,\"timestamp\":\"2023-11-29T07:48:43.020873Z\",\"winlog\":{\"channel\":\"Microsoft-Windows-Sysmon/Operational\",\"computer_name\":\"Windows10-WS1.acmeonions.com\",\"event_id\":1,\"opcode\":0,\"provider_guid\":\"5770385F-C22A-43E0-BF4C-06F5698FFBD9\",\"provider_name\":\"Microsoft-Windows-Sysmon\",\"record_id\":13741,\"task\":1,\"version\":5,\"process\":{\"pid\":2880,\"thread_id\":4400},\"event_data\":{\"CommandLine\":\"\\\"C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.18.2011.6-0\\\\MpCmdRun.exe\\\" -IdleTask -TaskName WdCacheMaintenance\",\"Company\":\"Microsoft Corporation\",\"CurrentDirectory\":\"C:\\\\Windows\\\\system32\\\\\",\"Description\":\"Microsoft Malware Protection Command Line Utility\",\"FileVersion\":\"4.18.2011.6 (WinBuild.160101.0800)\",\"Hashes\":\"MD5=D1DC475DC8A08618A40809F5F2CBC5E4,SHA256=FD00C4BA457AB1B207EDE405AB6CFBA2EE76D82C5936A10CA77D82CD5F0E7588,IMPHASH=F214FB46830FF51943E760685C3F8DA7\",\"Image\":\"C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Platform\\\\4.18.2011.6-0\\\\MpCmdRun.exe\",\"IntegrityLevel\":\"System\",\"LogonGuid\":\"36C8AC58-C64E-5FD3-E703-000000000000\",\"LogonId\":\"0x3e7\",\"OriginalFileName\":\"MpCmdRun.exe\",\"ParentCommandLine\":\"C:\\\\Windows\\\\system32\\\\svchost.exe -k netsvcs -p -s Schedule\",\"ParentImage\":\"C:\\\\Windows\\\\System32\\\\svchost.exe\",\"ParentProcessGuid\":\"36C8AC58-C64F-5FD3-1C00-000000001700\",\"ParentProcessId\":1200,\"ProcessGuid\":\"36C8AC58-3039-5FD4-B604-000000001700\",\"ProcessId\":7816,\"Product\":\"Microsoft® Windows® Operating System\",\"RuleName\":\"-\",\"TerminalSessionId\":0,\"User\":\"NT AUTHORITY\\\\SYSTEM\",\"UtcTime\":\"2023-11-29 07:48:43.020\",\"Status\":null}},\"log\":{\"file\":{\"name\":\"/tmp/data.evtx\"}},\"event\":{\"code\":1,\"created\":\"2023-11-29T07:48:43.020873Z\"},\"@timestamp\":\"2023-11-29T07:48:43.020873Z\"}", "@timestamp"=>2023-11-29T12:44:29.814Z, "type"=>"redis-input", "ecs"=>{"version"=>"8.0.0"}, "winlog"=>{"event_id"=>1, "opcode"=>0, "event_data"=>{"ProcessGuid"=>"36C8AC58-3039-5FD4-B604-000000001700", "TerminalSessionId"=>0, "Image"=>"C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2011.6-0\\MpCmdRun.exe", "LogonGuid"=>"36C8AC58-C64E-5FD3-E703-000000000000", "ParentProcessGuid"=>"36C8AC58-C64F-5FD3-1C00-000000001700", "CurrentDirectory"=>"C:\\Windows\\system32\\", "Hashes"=>"MD5=D1DC475DC8A08618A40809F5F2CBC5E4,SHA256=FD00C4BA457AB1B207EDE405AB6CFBA2EE76D82C5936A10CA77D82CD5F0E7588,IMPHASH=F214FB46830FF51943E760685C3F8DA7", "FileVersion"=>"4.18.2011.6 (WinBuild.160101.0800)", "ParentProcessId"=>1200, "UtcTime"=>"2023-11-29 07:48:43.020", "RuleName"=>"-", "ParentCommandLine"=>"C:\\Windows\\system32\\svchost.exe -k netsvcs -p -s Schedule", "Status"=>nil, "User"=>"NT AUTHORITY\\SYSTEM", "ProcessId"=>7816, "CommandLine"=>"\"C:\\ProgramData\\Microsoft\\Windows Defender\\platform\\4.18.2011.6-0\\MpCmdRun.exe\" -IdleTask -TaskName WdCacheMaintenance", "IntegrityLevel"=>"System", "ParentImage"=>"C:\\Windows\\System32\\svchost.exe", "Description"=>"Microsoft Malware Protection Command Line Utility", "OriginalFileName"=>"MpCmdRun.exe", "Product"=>"Microsoft® Windows® Operating System", "LogonId"=>"0x3e7", "Company"=>"Microsoft Corporation"}, "task"=>1, "process"=>{"thread_id"=>4400, "pid"=>2880}, "provider_name"=>"Microsoft-Windows-Sysmon", "channel"=>"Microsoft-Windows-Sysmon/Operational", "computer_name"=>"Windows10-WS1.acmeonions.com", "version"=>5, "provider_guid"=>"5770385F-C22A-43E0-BF4C-06F5698FFBD9", "record_id"=>13741}, "event_record_id"=>13741, "@version"=>"1", "timestamp"=>"2023-11-29T07:48:43.020873Z", "tags"=>["import", "elastic-agent", "input-so-manager", "beats_input_codec_plain_applied"], "input"=>{"type"=>"log"}, "import"=>{"file"=>"data.json", "id"=>"fff4261b8064add1ddbb546f4c59dece"}, "metadata"=>{"input"=>{"beats"=>{"host"=>{"ip"=>"172.17.1.1"}}}, "raw_index"=>"logs-import-so", "beat"=>"filebeat", "version"=>"8.10.4", "type"=>"_doc", "input_id"=>"logfile-logs-ac008d73-25c0-4ed5-a104-4876aa45d27f", "stream_id"=>"logfile-log.logs-ac008d73-25c0-4ed5-a104-4876aa45d27f", "pipeline"=>"logs-windows.sysmon_operational-1.24.0"}, "elastic_agent"=>{"snapshot"=>false, "id"=>"d9f14abd-247c-4981-894a-ff87ee155934", "version"=>"8.10.4"}, "data_stream"=>{"dataset"=>"import", "type"=>"logs", "namespace"=>"so"}, "cloud"=>{"region"=>"", "instance"=>{"id"=>"38268ba4-d95c-47f7-ac34-b0cbc716fd86"}, "provider"=>"huawei", "availability_zone"=>"nova", "service"=>{"name"=>"ECS"}}, "agent"=>{"version"=>"8.10.4", "id"=>"d9f14abd-247c-4981-894a-ff87ee155934", "type"=>"filebeat", "name"=>"so-manager", "ephemeral_id"=>"ac7bbe98-958d-4d60-a6f4-acdbcac892f6"}, "host"=>{"mac"=>["02-34-61-95-16-5F", "02-42-0E-FB-CC-A7", "02-42-9B-A0-58-3C", "0A-56-9F-DC-0B-6C", "0A-6F-8C-FE-D7-19", "26-A1-C3-A3-0C-39", "26-D9-04-56-9F-A1", "2A-4C-9D-01-05-17", "2A-E5-09-E4-3E-7C", "36-80-EA-2B-8E-3F", "52-CB-BD-49-BB-08", "6A-30-68-7C-1F-42", "6E-37-0C-12-A7-16", "72-55-C5-4B-83-73", "82-BF-CB-6A-2B-73", "86-8E-76-2C-A1-82", "8A-6E-95-70-8D-80", "96-1E-D7-C6-88-E5", "A2-51-7F-1F-D5-EC", "AE-5F-B8-32-AA-57", "AE-C9-21-B9-D6-9E", "B6-D1-7F-F1-CD-EC", "BE-B7-0C-17-16-1A", "E2-BA-27-EF-9C-00", "EA-50-B6-3F-DF-BA", "FA-16-3E-33-24-71", "FA-16-3E-94-9B-30"], "hostname"=>"so-manager", "id"=>"38268ba4d95c47f7ac34b0cbc716fd86", "name"=>"so-manager", "containerized"=>false, "architecture"=>"x86_64", "os"=>{"kernel"=>"5.15.0-101.103.2.1.el9uek.x86_64", "version"=>"9.3", "type"=>"linux", "name"=>"Oracle Linux Server", "family"=>"redhat", "platform"=>"ol"}, "ip"=>["192.168.75.15", "192.168.76.30", "172.17.0.1", "172.17.1.1"]}, "container"=>{"id"=>"data.json"}}], :response=>{"create"=>{"_index"=>"logs-import-so", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [logs-windows.sysmon_operational-1.24.0] does not exist"}}}}

include reproduction steps

• Perform baseline install
• Attempt to import .evtx to securityonion instance

[weslambert]

Confirmed working with latest changes.