Improve notable validation #34

pyth0n1c · 2023-07-31T17:13:51Z

The following PR ensures that the following validation takes places on observables during STATIC validation (contentctl validate):

Each observable (which could appear in the message in the format $name$ or in the declaration of an observable) appears as a string in the search itself. This check occurs during the validate phase
Each observable above which is called our in the format $name$ appears as a field in the results generated from executing the search. This check occurs during the test phase.

Finally, after contentctl test runs and a search is tested with data, we make sure that all the fields in the search appear, populated with values (the value of the field is not 'null') in the results of the search.

It also includes a large number of other changes:

When a search references a macro or lookup, we ensure that the macro/lookups exists. If not, we throw a descriptive error.
Some updates to the format of the contentctl_test.yml file to support more fine-grained control over the infrastructure tests are run against (server vs container).
The ability to pass the credentials for one or more servers on the command line. This way, credentials and addresses do NOT need to be hardcoded into the configuration file.

of having it be a dictionary. This allows for better validation and is more Pythonic. Also, validate that only valid observables are used in the message field of a detection and that every observable declared is used in the message field.

search as well for the notables that we have called out. A more thorough check will also be done after the search runs in contentctl test, but this static check will likely catch most simple issues.

when fields are missing after running a search with real data on a splunk instance. This helps determine to and even higher degree if notables were declared correctly and gives a high degree of certainty that they will be generated correctly in ES.

missing a notable field in the underlying search

by removing un-needed libraries

file

and message fields - these should be allowed to be independent. Still require that all fields called out in the message or in the observable are present in the search. This commit adds in the static checks.

for macros and lookups. Now, if we do not find a macro or a lookup instead of just ignoring it we will throw an error.

detections which have been affected by updated macros and/or lookups, to include lookup CSV and/or YML files.

by removing quotes.

a number of areas centered around the contentctl_test.yml configuration file and underlying object. This will support better for testing against multiple targets.

Don't override default test options unless CLI arguments are actually passed.

…l support for passing the address and credentials of servers on the command line.

or environment variable environments!

WORKS, but only produces a warning message. It is presently disabled due to a large amount of nonconformant content.

process when an appinspect throws any kind of errors/warnings/failures/ manual checks. All of these can prevent automatic approval in Splunkbase or deployment of an app.

pyth0n1c added 6 commits July 28, 2023 13:29

Create an Observable type instead

b81582a

of having it be a dictionary. This allows for better validation and is more Pythonic. Also, validate that only valid observables are used in the message field of a detection and that every observable declared is used in the message field.

Improve validation to check

db06257

search as well for the notables that we have called out. A more thorough check will also be done after the search runs in contentctl test, but this static check will likely catch most simple issues.

More enhancements to catch

7b3b5da

when fields are missing after running a search with real data on a splunk instance. This helps determine to and even higher degree if notables were declared correctly and gives a high degree of certainty that they will be generated correctly in ES.

Fixing template detection which was

ea5c86f

missing a notable field in the underlying search

Fix testEndToEndWorkflow

9e591a3

by removing un-needed libraries

Fix and update poetry.lock

dc9fa25

file

pyth0n1c added enhancement New feature or request WIP labels Jul 31, 2023

pyth0n1c mentioned this pull request Jul 31, 2023

Validation checks improvements #33

Closed

pyth0n1c linked an issue Jul 31, 2023 that may be closed by this pull request

Validation checks improvements #33

Closed

pyth0n1c added 16 commits August 1, 2023 10:04

Fix bug where lookup folder was not generated properly in app.

081fd7a

Do not require validation between observable

3838937

and message fields - these should be allowed to be independent. Still require that all fields called out in the message or in the observable are present in the search. This commit adds in the static checks.

Significantly improved validation

816f4f7

for macros and lookups. Now, if we do not find a macro or a lookup instead of just ignoring it we will throw an error.

Progress on implementing --mode changes

174a0c3

Better support for getting modified content

06dd1a4

Proper support for contentctl test checking

1c3a8a8

detections which have been affected by updated macros and/or lookups, to include lookup CSV and/or YML files.

Improve error message

c11bf16

by removing quotes.

Huge number of overhauls and updates in

e0d3b23

a number of areas centered around the contentctl_test.yml configuration file and underlying object. This will support better for testing against multiple targets.

Fixes to incorrect fields called for objects

8bf2d47

Clean up container when done.

29c1848

Don't override default test options unless CLI arguments are actually passed.

Fix typo in variable name

79b7fbd

Fix missing sid when there is an error.

32bb047

Better support for the --num_containers command line argument. Initia…

2ef4d89

…l support for passing the address and credentials of servers on the command line.

Should now accept command line

b770bb5

or environment variable environments!

Fix so that notable validation

dba2ad8

WORKS, but only produces a warning message. It is presently disabled due to a large amount of nonconformant content.

Throw errors during build

90717ee

process when an appinspect throws any kind of errors/warnings/failures/ manual checks. All of these can prevent automatic approval in Splunkbase or deployment of an app.

P4T12ICK mentioned this pull request Sep 15, 2023

Bring together the latest improvements #58

Merged

pyth0n1c merged commit 5421566 into main Oct 9, 2023
0 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve notable validation #34

Improve notable validation #34

pyth0n1c commented Jul 31, 2023 •

edited

Loading

Improve notable validation #34

Improve notable validation #34

Conversation

pyth0n1c commented Jul 31, 2023 • edited Loading

pyth0n1c commented Jul 31, 2023 •

edited

Loading