Update security service integrations packages mappings #12624

mrodm · 2025-02-05T16:38:07Z

Proposed commit message

Updated failed mappings found in system tests while validating the documents based on mappings in the data-streams ~~and the corresponding transforms~~.

Example of Buildkite build where the failures are raised: https://buildkite.com/elastic/integrations/builds/20794

Packages updated here:

box_events

[0] field "threat.enrichments.indicator.first_seen" is undefined: actual mapping type (keyword) does not match with ECS definition type: date
[1] field "threat.enrichments.indicator.last_seen" is undefined: actual mapping type (keyword) does not match with ECS definition type: date

claroty_ctd

[0] field "threat.indicator.modified_at" is undefined: actual mapping type (keyword) does not match with ECS definition type: date

crowdstrike

[0] field "crowdstrike.assessments.analytics_and_improvements_mac" is undefined: field definition not found
[1] field "crowdstrike.assessments.application_firewall_mac" is undefined: field definition not found
[2] field "crowdstrike.assessments.crendential_dumping_hash_mac" is undefined: field definition not found
...

mimecast

[0] field "threat.indicator.first_seen" is undefined: actual mapping type (keyword) does not match with ECS definition type: date
[1] field "threat.indicator.modified_at" is undefined: actual mapping type (keyword) does not match with ECS definition type: date

sublime_security

[0] field "sublime_security.email_message.headers.hops.fields.position" is undefined: field definition not found

ti_anomali

[0] field "threat.indicator.modified_at" is undefined: actual mapping type (keyword) does not match with ECS definition type: date

Packages with changes related to transforms moved to #12841

github (transforms)

field "message" is undefined: actual mapping type (keyword) does not match with ECS definition type: match_only_text

ti_custom (transforms)

[0] field "threat.indicator.url.original" is undefined: actual mapping type (keyword) does not match with ECS definition type: wildcard

tychon (transforms)

[0] field "related.ip" is undefined: actual mapping type (keyword) does not match with ECS definition type: ip
[1] field "server.ip" is undefined: actual mapping type (keyword) does not match with ECS definition type: ip

wiz (transforms)

[0] field "ecs.version" is undefined: actual mapping type (text) does not match with ECS definition type: keyword
[1] field "message" is undefined: actual mapping type (text) does not match with ECS definition type: match_only_text

[0] field "device.id" is undefined: actual mapping type (text) does not match with ECS definition type: keyword
[1] field "ecs.version" is undefined: actual mapping type (text) does not match with ECS definition type: keyword
[2] field "message" is undefined: actual mapping type (text) does not match with ECS definition type: match_only_text
[3] field "related.ip" is undefined: actual mapping type (text) does not match with ECS definition type: ip
[4] field "tags" is undefined: actual mapping type (text) does not match with ECS definition type: keyword
[5] field "vulnerability.reference" is undefined: actual mapping type (text) does not match with ECS definition type: keyword

Teleport changes moved to #12851

teleport (ingest pipeline event-groups)

[0] field "teleport.audit.account_id" is undefined: field definition not found
[1] field "teleport.audit.aws_host" is undefined: field definition not found
[2] field "teleport.audit.aws_region" is undefined: field definition not found
[3] field "teleport.audit.aws_service" is undefined: field definition not found
[4] field "teleport.audit.exit_code" is undefined: field definition not found
[5] field "teleport.audit.instance_id" is undefined: field definition not found
[6] field "teleport.audit.region" is undefined: field definition not found
[7] field "teleport.audit.status" is undefined: field definition not found
[8] field "teleport.audit.target" is undefined: field definition not found

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Add changelog entries for all the updated packages
Test with elastic-package enabled mappings https://buildkite.com/elastic/integrations/builds/21924
Is it required to update transform settings (destination index)?
Remove debug changes .buildkite folder and go.mod/go.sum files.
Remove unnecessary comments from package files.

How to test this PR locally

Run elastic-package with mappings validation enabled and run system tests:

cd packages/<package>
elastic-package stack up -v -d --version <kibana_constraint>
export ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings
# Add defer-cleanup to check the data ingested in the corresponding indices/data streams
elastic-package test system -v --defer-cleanup 900s

elastic-package stack down -v

Related issues

elastic-vault-github-plugin-prod · 2025-02-05T17:16:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

… - sublime_security

(cherry picked from commit fa96beb)

packages/box_events/data_stream/events/fields/ecs.yml

packages/claroty_ctd/data_stream/event/fields/ecs.yml

packages/crowdstrike/data_stream/fdr/fields/fields.yml

mrodm · 2025-02-06T18:42:53Z

packages/github/elasticsearch/transform/latest_code_scanning/transform.yml

@@ -10,7 +10,7 @@ source:
 # that ability in order to prevent having duplicate data and prevent query
 # time field type conflicts.
 dest:
-  index: "logs-github_latest.dest_code_scanning-1"
+  index: "logs-github_latest.dest_code_scanning-2"


If a new field definition is added to a transform, should this destination index be updated (increment suffix number)? Or keep that destination index without changes?

Same doubt for the other packages updating field definitions in transforms (tychon and wiz).

/cc @kcreddy

We increment the index to avoid any conflicts due to mapping changes.
I also simulated an integration upgrade from keyword to match_only_text on same index. After the upgrade, the type remained to be keyword and didn't change to match_only_text.

So, I guess this would mean that the destination index must be updated, am I right ? @kcreddy

In the other transforms (from other packages), there are changes related to field definitions with changes in mappings like:

keyword to ip

text to ip

text to keyword

text to match_only_text

keyword to wildcard

keyword to match_only_text

Should we update also the destination index there?

So, I guess this would mean that the destination index must be updated, am I right ?

@mrodm, yes the destination index version has to be updated

I've bumped the version defined in the fleet_transform_version setting for all the transforms that I've modified here @kcreddy

packages/github/elasticsearch/transform/latest_code_scanning/fields/ecs.yml

packages/sublime_security/data_stream/email_message/fields/fields.yml

mrodm · 2025-02-07T09:19:41Z

packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml

+- name: ecs.version
+  external: ecs


Error: field "ecs.version" is undefined: actual mapping type (text) does not match with ECS definition type: keyword

This change ensures that ecs.version is set as keyword.

mrodm · 2025-02-07T09:20:50Z

packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml

+- name: message
+  external: ecs


Error: field "message" is undefined: actual mapping type (text) does not match with ECS definition type: match_only_text

This change ensures that message is set as match_only_text.

Same query as above.

mrodm · 2025-02-07T09:21:53Z

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml

+- name: device.id
+  external: ecs


Error: field "device.id" is undefined: actual mapping type (text) does not match with ECS definition type: keyword

This change ensures that device.id is mapped as keyword.

mrodm · 2025-02-07T09:22:42Z

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml

+- name: vulnerability.reference
+  external: ecs


Error: field "vulnerability.reference" is undefined: actual mapping type (text) does not match with ECS definition type: keyword

This change ensures that vulnerability.reference is mapped as keyword.

mrodm · 2025-02-07T09:23:30Z

packages/wiz/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml

+- name: tags
+  external: ecs


Error: field "tags" is undefined: actual mapping type (text) does not match with ECS definition type: keyword

This change ensures that tags is mapped as keyword.

Add missing tychon

…s_mappings

mrodm · 2025-02-18T18:10:38Z

I was wondering if it would be worth to move all the changes related to transforms to another Pull Request so all the other packages can be updated (exception for teleport), WDYT ? @jsoriano @efd6
I'm not sure if there are any concerns related to the changes applied in the transforms here in this PR (e.g. this one #12624 (comment))

With the latest changes applied in elastic-package, validations in transforms are not going to be checked for now. The changes in those transforms were applied in this PR to take advantage of fixing them since they were detected here in this Buildkite build (https://buildkite.com/elastic/integrations/builds/20794).

efd6 · 2025-02-18T20:48:29Z

I was wondering if it would be worth to move all the changes related to transforms to another Pull Request so all the other packages can be updated (exception for teleport), WDYT ?

That wouldn't make me unhappy.

…s_mappings

mrodm · 2025-02-19T14:52:06Z

I was wondering if it would be worth to move all the changes related to transforms to another Pull Request so all the other packages can be updated (exception for teleport), WDYT ?

That wouldn't make me unhappy.

The changes of the packages that were updating transform field definitions have been moved to #12841 (github, ti_custom, tychon and wiz). I've tried to keep links to the previous discussions.

Updated the description accordingly.

efd6

LGTM after clarification of nits.

.buildkite/scripts/common.sh

go.mod

mrodm · 2025-02-20T11:25:10Z

Moved teleport changes to #12851

elastic-sonarqube · 2025-02-20T12:19:30Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-02-20T12:19:34Z

💚 Build Succeeded

Buildkite Build
Commit: 66de372

History

💔 Build #22585 failed 9283cd3
💚 Build #22534 succeeded a21024c
💚 Build #22521 succeeded 994e144
💚 Build #22312 succeeded 3ed832d
💚 Build #22203 succeeded 67bd2ba
💚 Build #22200 succeeded d18eaba

cc @mrodm

mrodm self-assigned this Feb 5, 2025

qcorporation force-pushed the main branch from eda4138 to f728ca7 Compare February 5, 2025 22:00

mrodm added 14 commits February 6, 2025 14:00

Add missing ECS mappings - box_events

01ba124

Add missing ECS mappings - claroty_ctd

001e81e

Avoid using dynamic templates for flattened objects - crowdstrike

a72f5b4

Add missing ECS mappings - mimecast

69d5f8a

Fix sublime_security.email_message.headers.hops.fields group mappings…

ee10501

… - sublime_security

Update event-groups ingest pipeline - teleport

612ce1f

Add missing ECS field in latest_code_scanning transform - github

1420345

Update destination index transform - github

ef021f2

Add missing ECS mappings - ti_anomali

d048cc2

Add mapping for threat.indicator.url.original in transform - ti_custom

0abc3c4

Add missing field mappings in transforms - tychon

0bcb54c

Add missing field mappings in transforms - wiz

f74e314

Add changelog entries

698dbe9

Update logstash owner in manifest

1469158

(cherry picked from commit fa96beb)

mrodm force-pushed the update_security-service-integrations_packages_mappings branch from 8ddd8ff to 1469158 Compare February 6, 2025 13:01

mrodm commented Feb 6, 2025

View reviewed changes

packages/box_events/data_stream/events/fields/ecs.yml Show resolved Hide resolved

mrodm commented Feb 6, 2025

View reviewed changes

packages/claroty_ctd/data_stream/event/fields/ecs.yml Show resolved Hide resolved

mrodm commented Feb 6, 2025

View reviewed changes

packages/crowdstrike/data_stream/fdr/fields/fields.yml Show resolved Hide resolved

mrodm commented Feb 6, 2025

View reviewed changes

packages/github/elasticsearch/transform/latest_code_scanning/fields/ecs.yml Outdated Show resolved Hide resolved

Remove non-working definition (commented) - sublime_security

a9736cc

mrodm commented Feb 7, 2025

View reviewed changes

packages/sublime_security/data_stream/email_message/fields/fields.yml Outdated Show resolved Hide resolved

mrodm commented Feb 7, 2025

View reviewed changes

mrodm added 10 commits February 12, 2025 11:30

Regenerate test data for ti_anomali

58077e4

Remove commented field definition in sublime_security

879aa33

Update README - ti_anomali

849a22e

Update transform settings github

e1ec321

Update transform settings ti_custom

ae20579

Update transform settings wiz

3fa66aa

Update transform settings tychon

d18eaba

Add missing tychon

Merge upstream/main into update_security-service-integrations_package…

67bd2ba

…s_mappings

Merge upstream/main into update_security-service-integrations_package…

7c3e7ed

…s_mappings

Update description for headers.hops.fields.position field

3ed832d

mrodm mentioned this pull request Feb 19, 2025

Update security service integrations packages transform mappings #12841

Open

10 tasks

mrodm added 6 commits February 19, 2025 15:34

Merge upstream/main into update_security-service-integrations_package…

118d694

…s_mappings

Revert changes in wiz - moved to elastic#12841

695555e

Revert changes in tychon - moved to elastic#12841

c5f2640

Revert changes in ti_custom - moved to elastic#12841

689eb49

Revert changes in github - moved to elastic#12841

c96630c

Update set of packages to test

994e144

Teleport - Ensure system tests wait to be ingestd all test docs/logs

a21024c

mrodm requested a review from efd6 February 19, 2025 18:16

efd6 reviewed Feb 19, 2025

View reviewed changes

.buildkite/scripts/common.sh Outdated Show resolved Hide resolved

go.mod Show resolved Hide resolved

efd6 approved these changes Feb 20, 2025

View reviewed changes

mrodm mentioned this pull request Feb 20, 2025

[teleport] Update event-groups ingest pipeline to manage cloud fields if already present #12851

Draft

7 tasks

mrodm added 2 commits February 20, 2025 12:25

Remove changes from teleport - moved to elastic#12851

9283cd3

Remove teleport from subset of packages to test

66de372

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update security service integrations packages mappings #12624

Update security service integrations packages mappings #12624

mrodm commented Feb 5, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 5, 2025 •

edited

Loading

mrodm Feb 6, 2025 •

edited

Loading

efd6 Feb 12, 2025

kcreddy Feb 12, 2025

mrodm Feb 12, 2025

kcreddy Feb 12, 2025

mrodm Feb 12, 2025

mrodm Feb 7, 2025

mrodm Feb 7, 2025

efd6 Feb 12, 2025

mrodm Feb 7, 2025

mrodm Feb 7, 2025

mrodm Feb 7, 2025

mrodm commented Feb 18, 2025

efd6 commented Feb 18, 2025

mrodm commented Feb 19, 2025 •

edited

Loading

efd6 left a comment

mrodm commented Feb 20, 2025 •

edited

Loading

elastic-sonarqube bot commented Feb 20, 2025

elasticmachine commented Feb 20, 2025

Update security service integrations packages mappings #12624

Are you sure you want to change the base?

Update security service integrations packages mappings #12624

Conversation

mrodm commented Feb 5, 2025 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

elastic-vault-github-plugin-prod bot commented Feb 5, 2025 • edited Loading

🚀 Benchmarks report

mrodm Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented Feb 18, 2025

efd6 commented Feb 18, 2025

mrodm commented Feb 19, 2025 • edited Loading

efd6 left a comment

Choose a reason for hiding this comment

mrodm commented Feb 20, 2025 • edited Loading

elastic-sonarqube bot commented Feb 20, 2025

Quality Gate passed

elasticmachine commented Feb 20, 2025

💚 Build Succeeded

History

mrodm commented Feb 5, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 5, 2025 •

edited

Loading

mrodm Feb 6, 2025 •

edited

Loading

mrodm commented Feb 19, 2025 •

edited

Loading

mrodm commented Feb 20, 2025 •

edited

Loading