TLS handshake errors in tap and tap-injector logs

[bwmetcalf]

What is the issue?

We are seeing TLS handshake error from ... errors in the tap and tap-injector logs:

~  % k get pod -n linkerd-viz -owide tap-845bb754c4-mn75s
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
tap-845bb754c4-mn75s   2/2     Running   0          14d   10.1.128.97   ip-10-1-137-171.us-west-2.compute.internal   <none>           <none>
~  % k logs -n linkerd-viz -c tap tap-845bb754c4-mn75s|grep handshake|head -10
2025/02/13 15:39:17 http: TLS handshake error from 10.1.128.97:51598: EOF
2025/02/13 15:39:17 http: TLS handshake error from 10.1.128.97:51576: EOF
2025/02/13 15:39:17 http: TLS handshake error from 10.1.128.97:51566: EOF
2025/02/13 15:39:17 http: TLS handshake error from 10.1.128.97:51582: EOF
2025/02/23 00:12:23 http: TLS handshake error from 10.1.128.97:43442: EOF
2025/02/23 00:12:23 http: TLS handshake error from 10.1.128.97:43438: EOF
2025/02/23 00:12:42 http: TLS handshake error from 10.1.128.97:60728: EOF
2025/02/23 00:12:42 http: TLS handshake error from 10.1.128.97:60718: EOF
2025/02/23 00:12:43 http: TLS handshake error from 10.1.128.97:60710: EOF
2025/02/23 00:17:08 http: TLS handshake error from 10.1.128.97:51188: EOF
~  % k get pod -n linkerd-viz -owide tap-injector-84899f676-gvwh8
NAME                           READY   STATUS    RESTARTS      AGE   IP             NODE                                        NOMINATED NODE   READINESS GATES
tap-injector-84899f676-gvwh8   2/2     Running   4 (23d ago)   33d   10.1.152.224   ip-10-1-154-46.us-west-2.compute.internal   <none>           <none>
~  % k logs -c tap-injector -n linkerd-viz tap-injector-84899f676-gvwh8|grep handshake|head -10
2025/02/04 18:30:00 http: TLS handshake error from 10.1.152.224:50512: EOF
2025/02/04 18:30:00 http: TLS handshake error from 10.1.152.224:50584: EOF
2025/02/04 19:30:00 http: TLS handshake error from 10.1.152.224:39828: EOF
2025/02/04 19:30:00 http: TLS handshake error from 10.1.152.224:39856: EOF
2025/02/04 19:45:00 http: TLS handshake error from 10.1.152.224:43334: EOF
2025/02/04 20:00:00 http: TLS handshake error from 10.1.152.224:60726: EOF
2025/02/04 20:00:00 http: TLS handshake error from 10.1.152.224:60740: EOF
2025/02/04 20:30:00 http: TLS handshake error from 10.1.152.224:54250: EOF
2025/02/04 20:45:00 http: TLS handshake error from 10.1.152.224:35714: EOF
2025/02/04 21:00:00 http: TLS handshake error from 10.1.152.224:50228: EOF

The IP address reported in the logs for each is the address assigned to the tape and tap-injector pods, respectively.

How can it be reproduced?

Presumably, any vanilla linkerd viz deployment will result in these errors.

Logs, error output, etc

See original description.

output of linkerd check -o short

~  % linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 24.11.8 but the latest edge version is 25.2.2
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running edge-24.11.8 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-84f8887748-b424m (edge-24.11.8)
	* linkerd-destination-84f8887748-dgbbw (edge-24.11.8)
	* linkerd-destination-84f8887748-wcs6l (edge-24.11.8)
	* linkerd-identity-66ff997c9-jlzf6 (edge-24.11.8)
	* linkerd-identity-66ff997c9-vt6wv (edge-24.11.8)
	* linkerd-identity-66ff997c9-z8q59 (edge-24.11.8)
	* linkerd-proxy-injector-5959bfcb57-8rddw (edge-24.11.8)
	* linkerd-proxy-injector-5959bfcb57-qt4hq (edge-24.11.8)
	* linkerd-proxy-injector-5959bfcb57-vr9q9 (edge-24.11.8)
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-84f8887748-b424m running edge-24.11.8 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/2.14/checks/#l5d-injection-disabled for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-76d9495fb5-5m896 (edge-24.11.8)
	* prometheus-554f465879-5p2v7 (edge-24.11.8)
	* tap-845bb754c4-mn75s (edge-24.11.8)
	* tap-injector-84899f676-gvwh8 (edge-24.11.8)
	* web-66f97d9494-7kb57 (edge-24.11.8)
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-76d9495fb5-5m896 running edge-24.11.8 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints

Status check results are √

Environment

% k version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.9-eks-8cce635

linkerd version is edge-24.11.8.

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None