Istio 1.21+ unable to get OS CA certificate bundle from SPIRE

[vassilvk]

• Version: 1.11.0
• Platform: Kubernetes 1.29.6 (EKS)
• Istio: 1.22
• Subsystem: server, agent, (istio?)

Not sure if this is an Istio or a SPIRE issue, however, after following this Istio + SPIRE guide, the Istio gateway pods detect the SPIRE-provided SDS API (as expected), however, they fail to obtain certificates from their SPIRE agents.

The gateway proxy logs contain a series of the following warning message:

2024-12-06T18:23:13.287724Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155	StreamSecrets gRPC config stream to sds-grpc closed: 3, workload is not authorized for the requested identities ["file-root:system"]	thread=28

The SPIRE agents log the following corresponding message:

time="2024-12-06T18:24:09Z" level=error msg="Error building stream secrets response" error="rpc error: code = InvalidArgument desc = workload is not authorized for the requested identities [\"file-root:system\"]" method=StreamSecrets pid=1364350 service=SDS.v3 subsystem_name=endpoints

The SPIRE infrastructure was deployed using spire helm chart version 0.24.1 (app version 1.11.0).

The ClusterSPIFFEID CRDs for the Istio gateway pods function correctly - I can see the SPIFFE registration entries for the gateway pods added to the SPIRE server.

What I don't see is an entry for identity file-root:system.

Not sure why the gateway is attempting to obtain a certificate for identity file-root:system, but looking at Istio's code this indicates that the proxy is asking for the OS root certificates.

I understand that this might be a an Istio question, but I figured someone here might have experienced the same.

[vassilvk]

Found out what causes the issue.

Istio 1.21 switched to verifying server certificates at proxies by default.

This leads to Istio proxies (both gateways and sidecars) trying to procure the operating system's CA by asking the SDS service for the file-root:system resource. When the SDS service is SPIRE, since no file-root:system registration entry exists on the SPIRE server, SPIRE agent fails to provide this secret, essentially breaking Istio's TLS verification and ability to originate HTTPS connections to non-mTLS endpoints.

Setting VERIFY_CERTIFICATE_AT_CLIENT env variable on istiod to false solves the issue for Istio 1.22 (note the docs incorrectly talk about VERIFY_CERT_AT_CLIENT).

The above workaround is not ideal as we need to have proxies verify TLS certificates coming from external servers.
Does it make sense to implement support for the file-root:system resource in SPIRE agent SDS implementation?

[evan2645]

Hi @vassilvk , thank you for opening this. I briefly read through the provided links .. can you confirm the exact Istio behavior change? Is it that the gateway previously handled TLS for connections to external services and then spoke MTLS back to the client ... and now, external connections can pass through and sidecars can directly validate TLS certificates presented by external services?

[vassilvk]

Hi @evan2645,

The difference in Istio behavior is caused by a change in the default value of the VERIFY_CERTIFICATE_AT_CLIENT istiod flag. Prior to 1.21, this flag was set to false. Starting with 1.21, the flag is set to true. When this flag is set to true, all Istio proxies in the mesh will attempt to validate server-side certificates during TLS (not mTLS) handshake.

The use case that leads to issues with SPIRE is as follows:

When the Istio proxy attempts to speak TLS (not mTLS) to an endpoint, when VERIFY_CERTIFICATE_AT_CLIENT for the control plane is set to true, the proxy checks if the DestinationRule for the endpoint includes CA certificates bundle (specified through the DestinationRule's spec.trafficPolicy.tls.caCertificates). If the CA cert bundle is provided through that setting, the proxy uses that to verify the server-provided certificate during the TLS handshake.

So that's all good.

If, however, the caCertificates setting of the DestinationRule is not provided (which is quite common), the proxy will attempt to use the operating system's CA bundle to verify server-side certificates.

When the proxy is configured to pull its secrets from an SDS stream, it will try to get the operating system's CA bundle by asking the SDS service (in our case SPIRE) for a resource called file-root:system. Since SPIRE has no such resource registered as a SPIFFE identity, nor does it recognize it as a well-known resource (the way it recognizes default and ROOTCA), it blows up.

My question is, does it make sense to add support for resource file-root:system to SPIRE agent, similarly to how it is done for default and ROOTCA and have the agent return its own operating system CA bundle when asked to provide this resource.

[evan2645]

Thank you @vassilvk - yes I think it will make sense to support this, just trying to make sure I fully understand first

When this flag is set to true, all Istio proxies in the mesh will attempt to validate server-side certificates during TLS (not mTLS) handshake.

When this flag is set to false as it was before .. what is the behavior? Gateway validates server-side cert on the client's behalf? Or something else?

[vassilvk]

When this flag is set to false as it was before.. what is the behavior?

When VERIFY_CERTIFICATE_AT_CLIENT is set to false at the control plane level, proxies in the mesh do not validate certificates for destinations whose CA certificate is not provided by the corresponding DestinationRule. At least that's my understanding.