default-network-dns-policy does not allow access to monitoring port TCP 9153

[Martin-Weiss]

Environmental Info:
RKE2 Version: 1.20.5

Cluster Configuration:
cis-1.5

Describe the bug:
Prometheus alerts show up for dns down.

Steps To Reproduce:
Deploy monitoring on an rke2 cluster with cis1.5 active / network policies active
Check alerts

Expected behavior:
No alert should show up in case dns is up and running well

Actual behavior:
Alert is there because prometheus check to dns pods on 9153 does not work / is blocked by network policy

Workaround:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-network-dns-monitoring-policy
  namespace: kube-system
spec:
  ingress:
  - ports:
    - port: 9153
      protocol: TCP
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  policyTypes:
  - Ingress

[brandond]

I believe this should go into rancher/rancher. RKE2 isn't responsible for shipping PSPs for every chart; PSPs should be bundled with the apps that require them to function.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[Martin-Weiss]

Not stale

[jayunit100]

hey was looking at some stuff in this area - and this is an interesting issue - in general, for rke2 do we apply default network policies according to a specification/where are the knobs for controlling them ?

[jayunit100]

cc @brandond ~ i guess you would know per the PSP stuff - specifically i wanted to see if i could add more default PSPs and NetworkPolicies to RKE2 clusters

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[QuinnBast]

This was never completed, unfortunately.

Also duplicated in #1383

Looking for a workaround to this... I'm not sure how I am supposed to get Prometheus metrics for coredns without this port being added to the network policy

[brandond]

As discussed above at #909 (comment) and at #1383 (comment) - the monitoring application (the thing doing the scraping) should include PSPs that allow its scrape traffic. It is NOT expected that application monitoring ports would be open to all pods by default, this would be a violation of the principle of least privilege.

I'm not sure how I am supposed to get Prometheus metrics for coredns without this port being added to the network policy

Add a network policy that allows prometheus to access the coredns metrics port? Why would you need to modify the existing one?