Send custodian logs to datadog

[sriram9707]

Hi,

We are enforcing compliance monitoring using cloud custodian. However we need to send these logs to datadog for creating a custom dashboards and Managed triggers.

Can someone assists on this please

[kapilt]

logs or metrics... send to log group, send from log group to datadog using log destinations. for metrics send to cloudwatch.. have datadog harvest metrics.

[sriram9707]

Hey kapil, thanks for the response.

how do we store the custodian findings (like compliant/ non-complaint) to log group?. Do we need to configure it in custodian policy ?.

we have a subscription filter for sending the logs from log group to datadog.
@kapilt

[kapilt]

which data dog api / product are you trying to integrate with ? there's a few dozen out there. custodian findings ? compliant / non compliant. those sound like very specific usages.. like security hub or config rules. could you please elaborate on what your trying to do, with policy example.

[sriram9707]

We are basically trying to send custodian logs from security hub( like resource tag is missing, ebs uncrypted ) to datadog in json format so that we can create a dashboard on similar findings. @kapilt

[sriram9707]

the logs which are sent to datadog from securityhub is of array of json data type. because of that we were unable to make use of these logs. thats why i'm figuring an other way to send these logs to datadog.

sample log for security hub in datadog:-

{"ProductArn":"arn:aws:securityhub:us-west-2::product/aws/securityhub","Types":["Software and Configuration Checks/Industry and Regulatory Standards/CIS AWS Foundations Benchmark"],"Description":"IAM policies are the means by which privileges are granted to users, groups, or roles. It is recommended and considered a standard security advice to grant least privilege—that is, granting only the permissions required to perform a task. Determine what users need to do and then craft policies for them that let the users perform only those tasks, instead of allowing full administrative privileges.","SchemaVersion":"2018-10-08","Compliance":{"Status":"PASSED"},"GeneratorId":"arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0/rule/1.22","FirstObservedAt":"2020-05-22T02:08:56.418Z","CreatedAt":"2020-05-22T02:08:56.418Z","RecordState":"ACTIVE","Title":"1.22 Ensure IAM policies that allow full ":" administrative privileges are not created","Workflow":{"Status":"RESOLVED"},"LastObservedAt":"2021-03-16T14:09:00.920Z","Severity":{"Normalized":0,"Label":"INFORMATIONAL","Product":0,"Original":"INFORMATIONAL"},"UpdatedAt":"2021-03-16T14:08:59.554Z","FindingProviderFields":{"Types":["Software and Configuration Checks/Industry and Regulatory Standards/CIS AWS Foundations Benchmark"],"Severity":{"Normalized":0,"Label":"INFORMATIONAL","Product":0,"Original":"INFORMATIONAL"}},"WorkflowState":"NEW","ProductFields":{"RelatedAWSResources:0/type":"AWS::Config::ConfigRule","RuleId":"1.22","RecommendationUrl":"https://docs.aws.amazon.com/console/securityhub/standards-cis-1.22/remediation","aws/securityhub/ProductName":"Security Hub","StandardsControlArn":"arn:aws:securityhub:us-west-2::control/cis-aws-foundations-benchmark/v/1.2.0/1.22","aws/securityhub/CompanyName":"AWS","StandardsGuideSubscriptionArn":"arn:aws:securityhub:us-west-2::subscription/cis-aws-foundations-benchmark/v/1.2.0","StandardsGuideArn":"arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0","RelatedAWSResources:0/name":"securityhub-iam-policy-no-statements-with-admin-access-737d6b0e","aws/securityhub/FindingId":"arn:aws:securityhub:us-west-2::product/aws/securityhub/arn:aws:securityhub:us-west-2::subscription/cis-aws-foundations-benchmark/v/1.2.0/1.22/finding/9ea28359-8482-4437-b9b8-a3ec059b66fb"},"AwsAccountId":"","Id":"arn:aws:securityhub:us-west-2::subscription/cis-aws-foundations-benchmark/v/1.2.0/1.22/finding/9ea28359-8482-4437-b9b8-a3ec059b66fb","Remediation":{"Recommendation":{"Text":"For directions on how to fix this issue, please consult the AWS Security Hub CIS documentation.","Url":"https://docs.aws.amazon.com/console/securityhub/standards-cis-1.22/remediation"}},"Resources":[{"Partition":"aws","Type":"AwsIamPolicy","Details":{"AwsIamPolicy":{"Path":"/","IsAttachable":true,"UpdateDate":"2020-03-11T11:50:40.000Z","PermissionsBoundaryUsageCount":0,"AttachmentCount":1,"PolicyName":"SSEventBridgeCrossAccPolicy","DefaultVersionId":"v1","PolicyVersionList":[{"VersionId":"v1","IsDefaultVersion":true,"CreateDate":"2020-03-11T11:50:40.000Z"}],"CreateDate":"2020-03-11T11:50:40.000Z","PolicyId":""}},"Region":"us-west-2","Id":"arn:aws:iam:::policySEventBridgeCrossAccPolicy"}]}

@kapilt we get custodian logs in same format.

[veenagurram]

@sriram9707 Did you find any solution for this case?