Add builtin:logmonitoring.log-storage-settings to CR and reconciler

[waodim]

Description

Ticket

With this it is possible to define ingest rule matchers in the dynakube. This configuration needs to be stored in settings (schema builtin:logmonitoring.log-storage-settings) in the scope of the current cluster ( = KubernetesClusterMEID).

This logic queries inside the logmonitoring reconciler all settings with this schemaID, and if there is none, we create new ones based on the defined ingest rule matchers in the dynakube (default is empty).

IF we query the settings and there are already some defined, I decided to add a condition to the dynakube that says exactly that.

How can this be tested?

Go to the Environment API v2 page in your tenant and try it out. It is in the Settings section. At first you'll need to enable com.compuware.apm.webuiff.config.core.hierarchy.resolution.pg.k8workload.pgwlhr.feature

Deploy a dynakube with a valid logmonitoring section for example:

  logMonitoring:
    ingestRuleMatchers:
    - attribute: "k8s.deployment.name"
      values:
        - "dynatrace"
        - "anothertest"
    - attribute: "k8s.container.name"
      values:
        - "testContainerName"
  templates:
      logMonitoring:
        imageRef:
          repository: us-central1-docker.pkg.dev/cloud-platform-207208/chmu/logmodule
          tag: latest

Wait a little bit then run in the environment API page the get query for the settings with the schema. See if it is created properly.

It should look like this:

{
  "items": [
    {
      "objectId": "vu9U3hXa3q0AAAABACpidWlsdGluOmxvZ21vbml0b3JpbmcubG9nLXN0b3JhZ2Utc2V0dGluZ3MAEktVQkVSTkVURVNfQ0xVU1RFUgAQMDFGNjZGMjkxRjBENjU0MAAkMzcxM2RjMDktMDYzNC0zMjM0LTlhNTAtYTg4MTA4OGY2YmI0vu9U3hXa3q0",
      "value": {
        "enabled": true,
        "config-item-title": "dynakube-test",
        "send-to-storage": true,
        "matchers": [
          {
            "attribute": "k8s.deployment.name",
            "operator": "MATCHES",
            "values": [
              "dynatrace",
              "anothertest"
            ]
          },
          {
            "attribute": "k8s.container.name",
            "operator": "MATCHES",
            "values": [
              "testContainerName"
            ]
          }
        ]
      }
    }
  ],
  "totalCount": 1,
  "pageSize": 100
}

You can then grab the objectID, delete the setting and reapply a dynakube with an empty logmonitoring section and see if default settings gets created.

That looks like this:

[
  {
    "schemaId": "builtin:logmonitoring.log-storage-settings",
    "scope": "KUBERNETES_CLUSTER-D3A3C5A146830A79",
    "value": {
      "enabled": true,
      "config-item-title": "my-cluster-name",
      "send-to-storage": true,
      "matchers": []
    }
  }
]

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 43.95161% with 139 lines in your changes missing coverage. Please review.

Project coverage is 64.52%. Comparing base (50ecbca) to head (9ed8f4d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
test/mocks/pkg/clients/dynatrace/client.go	29.41%	36 Missing and 12 partials ⚠️
pkg/clients/dynatrace/settings.go	0.00%	28 Missing ⚠️
pkg/clients/dynatrace/settings_logmonitoring.go	64.17%	16 Missing and 8 partials ⚠️
...a3/dynakube/logmonitoring/zz_generated.deepcopy.go	0.00%	19 Missing and 1 partial ⚠️
...ynakube/logmonitoring/logmonsettings/reconciler.go	69.56%	10 Missing and 4 partials ⚠️
...g/controllers/dynakube/logmonitoring/reconciler.go	40.00%	2 Missing and 1 partial ⚠️
test/mocks/cmd/remote_command/executor.go	0.00%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4053      +/-   ##
==========================================
- Coverage   64.69%   64.52%   -0.17%     
==========================================
  Files         397      400       +3     
  Lines       26512    26754     +242     
==========================================
+ Hits        17151    17263     +112     
- Misses       8031     8136     +105     
- Partials     1330     1355      +25     

Flag	Coverage Δ
unittests	64.52% <43.95%> (-0.17%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[0sewa0]

settings are created and only created once, so works

but we are constantly (ie.: every reconcile) check if the settings exist this is a bit of an overkill

[aorcholski]

I removed ingestRuleMatchers from logMonitoring section to get empty logMinitoring: {}. Then I updated my dynakube and had to wait ~9 minutes for the default matcher to be added to the settings object on the server.
Is it OK? I guess we need to limit the number of queries we sent to the server. On the other hand it is a little bit annoying.

[waodim]

I removed ingestRuleMatchers from logMonitoring section to get empty logMinitoring: {}. Then I updated my dynakube and had to wait ~9 minutes for the default matcher to be added to the settings object on the server. Is it OK? I guess we need to limit the number of queries we sent to the server. On the other hand it is a little bit annoying.

Just a sidenote which is important IMO: we not only removed the ingestRuleMatchers in the dynakube but we also removed the settings in the Environment API manually while the condition was stored in the dynakube that there are already settings, then edited the dynakube -> this led to the waiting time. IMO i think this is ok, as I do not think it should be a normal use case to delete the settings in the API manually while everything is up and running. But maybe @chrismuellner has more context there and knows more.

[chrismuellner]

Operator currently has no good way of reacting to changes done on the Dynatrace API side. We are listening and reacting to changes in the Kubernetes cluster and need to find a way to better/quicker react to external changes.

This is not something we can quickly fix here, but we'll need to properly think about options we have. For the initial feature the delay is acceptable.

[aorcholski]

By the way could you please move log.Info("start reconciling app injection") to a line just before err = controller.injectionReconcilerBuilder(...).Reconcile(ctx) in pkg/controllers/dynakube.controller.go ?

	log.Info("start reconciling app injection")

	log.Info("start reconciling LogMonitoring")

	logMonitoringReconciler := controller.logMonitoringReconcilerBuilder(controller.client, controller.apiReader, dynatraceClient, dk)

[aorcholski]

LGTM