Encrypt traffic on the data-path "natively"

[markusthoemmes]

In what area(s)?

/area networking

Describe the feature

Currently, whenever we're asked on how the traffic inside of Knative can be secured, our default answer (I think) is: Use a service mesh and enable mTLS.

I guess that works in theory, but it has a ton of drawbacks coming with it. Among others: It forces the additional sidecar everywhere, which a) increases overhead and b) makes our data-plane actively worse as the activator has to deal with non-pod-addressability in the mesh case.

We also had a few users (read customers) ask specifically for a solution to encrypt traffic on the data-path without having to buy into the increased overhead that a mesh brings with it.

So: Can and should we solve this use-case "natively" within Knative? Personally, I think we should.

---

Thinking about potential designs for this, a few key questions come to mind:

1. What level of granularity is needed? Can we just configure a global certificate, serve that in all our components and call it a day? Do we need certs per-namespace or even per-service to secure the communication to the queue-proxy?
2. Do we need passthrough TLS, i.e. the ability to pass through encrypted traffic and let the application decrypt it (more on that below)
3. Do we need mTLS? Or are we fine with just TLS?
4. What kinda UX would this be? Do we ask people to muck with Secrets directly? Do we create a new API?

On passthrough encryption

Passthrough encryption certainly would be the best escape hatch here, as it'd allow us to defer this completely to the user's application. However, I don't believe we can implement that with our current assumptions. We need to have the ability to change headers of a request to allow for our routing to work. Even if we solved that, we need to be able to count requests for our autoscaling to work. Encrypted traffic wouldn't allow us to do either AFAIK (not an expert, more than happy to be wrong here).

[evankanderson]

A few quick thoughts/answers:

1. It's probably desirable to separate the certificates for system components from components that run in the user (developer) namespace (i.e. different certs for ingress/activator than for queue-proxy). Components that run in a developer namespace should be assumed to be available to all workloads (Knative and otherwise) in that namespace.

   In addition, we have two data paths to protect, one of which is managed by KIngress / multiple HTTP implementations, including non-Envoy implementations. This may restrict our ability to be especially creative (e.g. Client-cert-only TLS, which would be out of spec).

2. I think you're correct that we'd need to a lot of heavy lifting and cooption of the TLS protocol to get the desired semantics and support passthrough encryption. One option would be to have passthrough encryption but include the negotiated secrets passed through to the client.

   A simple thought experiment shows that we probably need to be able to crack the TLS request at the routing layer -- when the client connects to the ingress, they may not have provided a hostname until after ServerHello; this means that the ingress cannot scale-up-from-zero the correct backend until after the initial TLS handshake.

3. We may not mTLS if we can trust the Kubernetes network routing layer within the cluster (possibly a reasonable assumption). In that case, we could potentially distribute a shared "not a secret" certificate which supports perfect forward secrecy to all queue-proxies, and have them validate the "client certificate" from the ingress or activator.

   This should probably get at least a once-over from someone with more crypto experience than myself, who will probably yell "armchair cryptographer" at me and throw things.

4. It would be nice if we could manage the certificates (and any necessary secrets) ourselves, distributing them between the KIngress, activator, and queue-proxy. Alternatively, it would be nice if we could hook into a cert infrastructure like SPIFFE if it was available on the cluster; I'm not sure how difficult it would be to support both code paths.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[markusthoemmes]

/lifecycle frozen

[evankanderson]

This looks like it will be solved by #11906

/close

[knative-prow-robot]

@evankanderson: Closing this issue.

In response to this:

This looks like it will be solved by #11906

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.