ACLs in cluster broken when installed by helm

[japhar81]

Overview of the Issue

When installing a 3-node cluster via helm, and setting manageSystemACLs: true, anonymous tokens are used and produce errors.

---

Reproduction Steps

helm install --set acls.manageSystemACLs=true --set server.replicas=3

Observe logs in the resulting pods:

2024-10-03T19:21:51.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:35520 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:22:06.593Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:49148 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:22:21.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:33454 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2

-->

Consul info for both Client and Server

N/A, everything is defaulted from the chart

Operating system and Environment details

OKD 4.17 K8s cluster, bone-stock.

Log Fragments

2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/proxyconfiguration
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinations
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/httproute
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/catalog/v2beta1/workload
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/auth/v2beta1/partitiontrafficpermissions
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/tcproute
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/destinationpolicy
2024-10-03T19:18:58.799Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/mesh/v2beta1/meshconfiguration
2024-10-03T19:18:58.799Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2024-10-03T19:18:58.799Z [INFO] agent: Started gRPC listeners: port_name=grpc address=[::]:8502 network=tcp
2024-10-03T19:18:58.799Z [INFO] agent: started state syncer
2024-10-03T19:18:58.799Z [INFO] agent: Consul agent running!
2024-10-03T19:18:58.799Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2024-10-03T19:18:58.799Z [INFO] agent: Joining cluster...: cluster=LAN
2024-10-03T19:18:58.799Z [INFO] agent: (LAN) joining: lan_addresses=["consul-consul-server.consul.svc:8301"]
2024-10-03T19:18:58.844Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-consul-server-2 10.131.0.126
2024-10-03T19:18:58.845Z [INFO] agent.server: Adding LAN server: server="consul-consul-server-2 (Addr: tcp/10.131.0.126:8300) (DC: mgmt-0001)"
2024-10-03T19:18:58.847Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-consul-server-2.mgmt-0001 10.131.0.126
2024-10-03T19:18:58.847Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-consul-server-2.mgmt-0001 area=wan
2024-10-03T19:18:58.893Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-consul-server-1 10.129.2.120
2024-10-03T19:18:58.894Z [INFO] agent.server: Adding LAN server: server="consul-consul-server-1 (Addr: tcp/10.129.2.120:8300) (DC: mgmt-0001)"
2024-10-03T19:18:58.894Z [INFO] agent: (LAN) joined: number_of_nodes=3
2024-10-03T19:18:58.894Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
2024-10-03T19:18:58.897Z [INFO] agent.server: Found expected number of peers, attempting bootstrap: peers="10.128.2.123:8300,10.131.0.126:8300,10.129.2.120:8300"
2024-10-03T19:18:58.901Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-consul-server-1.mgmt-0001 10.129.2.120
2024-10-03T19:18:58.901Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-consul-server-1.mgmt-0001 area=wan
2024-10-03T19:19:06.136Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
2024-10-03T19:19:07.548Z [INFO] agent.server: New leader elected: payload=consul-consul-server-2
2024-10-03T19:19:08.242Z [WARN] agent: Node info update blocked by ACLs: node=e5639b5f-3d15-2792-59dd-af68731a1b31 accessorID="anonymous token"
2024-10-03T19:19:10.073Z [INFO] agent: Updated agent's ACL token: token=agent
2024-10-03T19:19:12.969Z [INFO] agent: Synced node info
2024-10-03T19:19:14.594Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:37500 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:29.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:35436 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:44.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:54786 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:19:59.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:42874 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."
2024-10-03T19:20:14.592Z [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=10.128.2.2:34498 error="Permission denied: anonymous token lacks permission 'agent:read' on \"consul-consul-server-0\". The anonymous token is used implicitly when a request does not specify a token."