[profiles] Do not error if pod needing deletion is not found

[khewonc]

What does this PR do?

Occasionally, when we need to delete existing agent pods that don't match a node's affinity, there's a chance that some pods are already deleted between the time the cached list of pods the operator queries is updated and the operator sends the deletion request. When that happens, the operator receives a pod not found error from the deletion request. Instead of causing a reconciler error, this PR ignores the not found error.

Motivation

CECO-1383

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

This PR is difficult to test since the issue only happens sporadically. It was reported when a new DatadogAgentProfile was created that triggered the operator to delete existing agent pods for not meeting the updated nodeaffinity requirements and affected approximately 1 out of 8 agent pods in that instance.

One way to test could be to create a DatadogAgentProfile in staging and ensuring you don't see a reconcile error log similar to the following during rollout of the profile:

{"level":"ERROR","ts":"2024-08-08T19:46:12.565Z","msg":"Reconciler error","controller":"datadogagent","controllerGroup":"datadoghq.com","controllerKind":"DatadogAgent","DatadogAgent":{"name":"datadog-agent","namespace":"datadog-agent"},"namespace":"datadog-agent","name":"datadog-agent","reconcileID":"xxx","error":"pods \"datadog-agent-xxxxx-xxxxx\" not found"}

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 49.08%. Comparing base (87f8143) to head (c9656fc).

Files with missing lines	Patch %	Lines
...troller/datadogagent/controller_reconcile_agent.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1608      +/-   ##
==========================================
+ Coverage   48.94%   49.08%   +0.14%     
==========================================
  Files         227      227              
  Lines       20636    20636              
==========================================
+ Hits        10101    10130      +29     
+ Misses      10010     9978      -32     
- Partials      525      528       +3     

Flag	Coverage Δ
unittests	49.08% <0.00%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...troller/datadogagent/controller_reconcile_agent.go	61.43% <0.00%> (+7.98%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87f8143...c9656fc. Read the comment docs.